https://redmine.pfsense.org/https://redmine.pfsense.org/favicon.ico?16780521162016-05-16T04:47:14ZpfSense bugtrackerpfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=271402016-05-16T04:47:14ZChris Buechlercbuechler@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Affected Version</strong> deleted (<del><i>2.3.1</i></del>)</li></ul><p>that's 2.3(.0)_1 rather than 2.3.1. It wasn't 2.3->2.3_1 that did it, since that only upgraded ntpd, rather something that would have happened on 2.3 as well. I'm guessing it's one of two things. Either something related to <a class="issue tracker-1 status-3 priority-5 priority-high4 closed" title="Bug: pkg update checking with no Internet access kills web GUI (Resolved)" href="https://redmine.pfsense.org/issues/6177">#6177</a>, or there seems to be some kind of issue with the IPsec dashboard widget causing that to happen for a few people.</p>
<p>If this is replicable for you, if you have the IPsec dashboard widget enabled, please try to remove that and see if that fixes the problem. That'll at least tell us where the issue resides.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=271592016-05-17T11:26:21ZBrent Kerlin
<ul></ul><p>Chris Buechler wrote:</p>
<blockquote>
<p>that's 2.3(.0)_1 rather than 2.3.1. It wasn't 2.3->2.3_1 that did it, since that only upgraded ntpd, rather something that would have happened on 2.3 as well. I'm guessing it's one of two things. Either something related to <a class="issue tracker-1 status-3 priority-5 priority-high4 closed" title="Bug: pkg update checking with no Internet access kills web GUI (Resolved)" href="https://redmine.pfsense.org/issues/6177">#6177</a>, or there seems to be some kind of issue with the IPsec dashboard widget causing that to happen for a few people.</p>
<p>If this is replicable for you, if you have the IPsec dashboard widget enabled, please try to remove that and see if that fixes the problem. That'll at least tell us where the issue resides.</p>
</blockquote>
<p>I have seen this issue frequently on clients since 2.3 rolled. I was more concerned with <a class="issue tracker-1 status-3 priority-11 priority-high3 closed" title="Bug: Interface dies with IPsec and SMP (Resolved)" href="https://redmine.pfsense.org/issues/6296">#6296</a> which was causing me many headaches, but I will try removing the IPSec widget on a few sites and report back (I have one with the webgui locked up right now who is a prime candidate). Any log dumps that would be helpful?</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=271602016-05-17T11:29:20ZBrent Kerlin
<ul></ul><blockquote>
<p>Restarting the webconfigurator from the console does not resolve the issue.<br />Other than the web not functioning, the firewall is performing as normal.</p>
</blockquote>
<p>Try restarting PHP-FPM from the console. That seems to clear up the issue for me...</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=272012016-05-20T09:56:44ZBrent Kerlin
<ul></ul><p>Brent Kerlin wrote:</p>
<blockquote>
<p>I have seen this issue frequently on clients since 2.3 rolled. I was more concerned with <a class="issue tracker-1 status-3 priority-11 priority-high3 closed" title="Bug: Interface dies with IPsec and SMP (Resolved)" href="https://redmine.pfsense.org/issues/6296">#6296</a> which was causing me many headaches, but I will try removing the IPSec widget on a few sites and report back (I have one with the webgui locked up right now who is a prime candidate). Any log dumps that would be helpful?</p>
</blockquote>
<p>I have removed the IPSec Widget from all the sites at which I was having this PHP-FPM issue. I'll report back in a couple days or if the problem persists.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=272202016-05-22T17:29:35ZRick Strangman
<ul></ul><p>I have no issues since removing the IPsec widget. Now on 2.3.1 and have not seen a lockup</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=272542016-05-24T12:48:59ZChris Buechlercbuechler@gmail.com
<ul><li><strong>Subject</strong> changed from <i>pfsense webconfigurator</i> to <i>IPsec dashboard widget causes GUI failure</i></li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Confirmed</i></li><li><strong>Target version</strong> set to <i>2.3.2</i></li><li><strong>Affected Version</strong> set to <i>2.3.x</i></li><li><strong>Affected Architecture</strong> <i></i> added</li><li><strong>Affected Architecture</strong> deleted (<del><i>amd64</i></del>)</li></ul> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=272562016-05-24T13:05:54ZAnonymous
<ul></ul><p>I have looked through the code again and nothing really stands out.</p>
<p>It would be helpful to know:</p>
<ul>
<li>How many tunnels do people have in cases where the issue is seen?</li>
<li>Does it make any difference if the widget is set to show Overview, Tunnels, or Mobile?</li>
</ul>
<p>THanks!</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=272572016-05-24T13:11:10ZChris Buechlercbuechler@gmail.com
<ul></ul><p>Steve Beaver wrote:</p>
<blockquote>
<p>I have looked through the code again and nothing really stands out.</p>
</blockquote>
<p>Ditto. Heard of roughly a handful of reports of this, but never seen it myself. Additional details would be appreciated.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=272642016-05-24T22:09:38ZChris Buechlercbuechler@gmail.com
<ul></ul><p>Thanks to Alex for getting me into an affected system. It's occasionally getting stuck in pfSense_ipsec_list_sa, without triggering any of the printfs there.</p>
<pre>
PHP_FUNCTION(pfSense_ipsec_list_sa) {
vici_conn_t *conn;
vici_req_t *req;
vici_res_t *res;
array_init(return_value);
vici_init();
conn = vici_connect(NULL);
if (conn) {
if (vici_register(conn, "list-sa", build_ipsec_sa_array, (void *) return_value) != 0) {
php_printf("VICI registration failed: %s\n", strerror(errno));
} else {
req = vici_begin("list-sas");
res = vici_submit(req, conn);
if (res) {
vici_free_res(res);
}
}
vici_disconnect(conn);
} else {
php_printf("VICI connection failed: %s\n", strerror(errno));
}
vici_deinit();
}
</pre>
<p>What I committed on this ticket should prevent this (and many other possible failure scenarios with commands that don't return) from killing the GUI. request_terminate_timeout will kill them off after 900 seconds. It only happens once every few minutes when continually refreshing a page that uses that function, so that's been enough to keep Alex's system from killing the GUI again.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=280452016-07-08T02:15:25ZChris Buechlercbuechler@gmail.com
<ul><li><strong>Target version</strong> changed from <i>2.3.2</i> to <i>2.4.0</i></li></ul> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=300722016-12-22T14:47:10ZJim Pingle
<ul><li><strong>File</strong> <a href="/attachments/1926">php-stuck-truss-04.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1926/php-stuck-truss-04.txt">php-stuck-truss-04.txt</a> added</li></ul><p>This also affects Status > IPsec</p>
<p>We have access to a customer system that has 70 tunnels defined, and it happens every 5-20 minutes (timing varies) while a browser is left on Status > IPsec. The requests are not piling up, they only take about 300ms to complete. Leaving a browser open on Status > IPsec with firebug or similar running, it's easy to spot when it stops responding.</p>
<p>When it happens, there are always two PHP child processes:</p>
<pre>
: ps uxawww | grep '[p]hp'
root 64113 0.5 0.9 272496 38300 - S 1:43PM 0:02.26 php-fpm: pool nginx (php-fpm)
root 267 0.0 0.6 268400 25140 - Ss 3:01AM 0:02.49 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root 64043 0.0 0.9 285304 38604 - I 1:43PM 0:00.19 php-fpm: pool nginx (php-fpm)
</pre>
<p>Attempting to run a truss on the top process (In state "S", sleeping) shows no output at all</p>
<p>Running truss on the other process (In state "I", idle) outputs info and then the browser gets a response. So long as the truss happens before the browser times out, everything keeps running. The truss output is attached. I have several more copies of truss output from other times I reproduced the issue, but they are all very close if not identical. I find it odd that merely attaching to the process with truss is somehow waking it up and causing it to proceed. I've tried hitting the process with other signals like <code>kill -HUP</code> but so far nothing brings it back to life but touching it with truss, or killing/restarting PHP-FPM.</p>
<p>There isn't much that happens in the AJAX request being made for Status > IPsec or the IPsec widget, it could be getting stuck in vici interaction.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=302462017-01-01T17:56:28ZJim Thompsonjim@netgate.com
<ul><li><strong>Assignee</strong> set to <i>Anonymous</i></li></ul> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=317552017-02-22T13:05:58ZNick Wenos
<ul></ul><p>We are also having what appears to be the same issue running on version 2.3.2 As a side affect of php-fpm going down our OpenVPN clients also lose the ability to connect until we restart php-fpm and openvpn. I don't know if this would affect all OpenVPN or just those using ssl cert authentication as is the case with our setup.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=317682017-02-23T08:25:42ZEric Machabertemachabert@sqli.com
<ul></ul><p>Nick Wenos wrote:</p>
<blockquote>
<p>We are also having what appears to be the same issue running on version 2.3.2 As a side affect of php-fpm going down our OpenVPN clients also lose the ability to connect until we restart php-fpm and openvpn. I don't know if this would affect all OpenVPN or just those using ssl cert authentication as is the case with our setup.</p>
</blockquote>
<p>We are also seeing this on 2.3.3<br />Running netstat -an shows request filling up the Recv-Q for IPC connection to /var/run/php-fpm.socket.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=324922017-04-10T09:17:04ZChris Bakercmbaker82@gmail.com
<ul></ul><p>I am also seeing this on 2.3.3. Is there any known work around other than removing the ipsec widget? Maybe changing the polling frequency?</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=332212017-07-13T11:50:04ZMarcio Merlonemmerlone@gmail.com
<ul></ul><p>I think this bug's priority should be raised since it also breaks openvpn functionality.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=333132017-07-26T09:56:12ZAnonymous
<ul><li><strong>Target version</strong> changed from <i>2.4.0</i> to <i>2.4.1</i></li></ul> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=334112017-08-09T11:08:56ZAnonymous
<ul><li><strong>Status</strong> changed from <i>Confirmed</i> to <i>Feedback</i></li><li><strong>Target version</strong> changed from <i>2.4.1</i> to <i>2.4.0</i></li></ul><p>I have done a LOT of research into this. I believe that since most dashboard widgets have their own timer, their own buffer and their own AJAX calling functions, they are from time to time stepping on each other and causing havoc on the server side.</p>
<p>As an experiment (for now) I have removed all of the individual refresh stuff from the widgets and replaced them with a single, central refresh service that loops though the dashboard updating each widget one at a time.</p>
<p>So far, the results appear to be dramatically better. I can't guarantee that this will solve the IPSec widget issue, but I think it might. I note that the time taken to refresh the IPSec widget has reduced from 5 seconds to about 10 mS so that has got to help.</p>
<p>The changes will be in 2.4-BETA later today.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=334122017-08-09T11:12:47Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Sounds like a fantastic change. Thanks Steve</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=339012017-09-15T14:38:46ZJim Pingle
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>New</i></li><li><strong>Target version</strong> changed from <i>2.4.0</i> to <i>2.4.1</i></li></ul><p>I still see this but it seems less common than it did in the past. Either have bad timing or sit on the dashboard too long with the IPsec widget and it still wedges.</p>
<p>Kicking it forward since it isn't critical.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=344672017-10-19T15:07:17ZJim Pingle
<ul><li><strong>Target version</strong> changed from <i>2.4.1</i> to <i>2.4.2</i></li></ul><p>There have been some IPsec widget fixes here which may be relevant, since it is so difficult to reproduce, it is difficult to know that it has been fully resolved. Moving forward.</p> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=346032017-10-23T12:19:18ZJim Pingle
<ul><li><strong>Target version</strong> changed from <i>2.4.2</i> to <i>2.4.3</i></li></ul> pfSense - Bug #6318: IPsec dashboard widget causes GUI failurehttps://redmine.pfsense.org/issues/6318?journal_id=356092018-01-16T14:05:10ZJim Pingle
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Affected Architecture</strong> <i>All</i> added</li><li><strong>Affected Architecture</strong> deleted (<del><i></i></del>)</li></ul><p>This appears to be fixed by other changes to the IPsec status code in recent versions. No new reports of this being caused by IPsec in some time.</p>