pfSense

+1; seems pretty replicable here when you leave the dashboard page open in a browser for a couple of hours. (Not 2.3.1 specific, was there with 2.3.0 as well.) Note that restarting the webserver alone (console option 11) does not help at all.

Well, while the original issue with the dashboard seems indeed gone, I managed to make the GUI completely unresponsive when upgrading pfBlockerNG package on two different boxes (2.3.1_1 full install, amd64). After the upgrade completed, the GUI did not return until PHP-FPM restart.

I am now experiencing this issue on 2.3.1-RELEASE-p1 (i386)

Ive been having intermittent Layer 7 DDoS Attacks for a day or few now, and it seems that after each wave of the attack the web configurator is only returning 502s from inside or outside.

Restarting web configurator does nothing, i have to restart php-fpm to fix the issue.

the flood causing this is the recent WordPress pingback amplification attack exploit, i have mitigated the flooding nodes via cloudflare now, but it would be nice if the web configurator didnt go all 502 error every time a new set of IPs tries the same attack....

Kill Bill wrote:

Well, while the original issue with the dashboard seems indeed gone, I managed to make the GUI completely unresponsive when upgrading pfBlockerNG package on two different boxes (2.3.1_1 full install, amd64). After the upgrade completed, the GUI did not return until PHP-FPM restart.

By any chance, did you use the "view" button in the Update Tab? Something has recently changed that is affecting that button's "End View" function...

BBcan177 . wrote:

By any chance, did you use the "view" button in the Update Tab? Something has recently changed that is affecting that button's "End View" function...

Yeah, not sure it's related to this issue but I noticed that button got broken as well.

Confirming this has happenned again on my unit, and this time without any attacks having hit us, i just had to restart the FPM again.

Xander Venterus wrote:

Confirming this has happenned again on my unit, and this time without any attacks having hit us, i just had to restart the FPM again.

Yeah, seen again multiple times on multiple 2.3.1_1 boxes.

FYI - Still happening on 2.3.2-RELEASE-p1 systems.

Sorry to re-hash this, but since it has just been assigned to me I need an update.

Some of the above responses would indicate this issue is PfBlockerNG specific. Is that the case? Is the problem present if pfB is not installed/active?

There is no known consistent single cause. Some have it with nothing else installed, some other pfBlocker, some with the IPsec widget, others hit it during HA XMLRPC sync. It's possible there is one root cause, or several, but so far it's not been simple to reproduce reliably under controlled conditions.

Well, to me it started to happen when I readded the "Traffic Graphs" widget. It never happened before without that.

We upgraded 2 days ago from 2.2.x to 2.3.1p1.
Same issue and no pfB installed.
We have upgraded only one of our two pfsense (the 2.2.x is halted) so we have many sync errors.

List of installed packages:
- Cron
- freeradius2
- Lightsquid
- nmap
- nrpe
- openvpn-client-export
- snort
- squid
- squidGuard

Steve Beaver wrote:

Sorry to re-hash this, but since it has just been assigned to me I need an update.

Some of the above responses would indicate this issue is PfBlockerNG specific. Is that the case? Is the problem present if pfB is not installed/active?

I just ran into this myself. Strangely, this issue causes all clients who try to connect with OpenVPN to just hang indefinitely. Restarting Web Configurator & PHP fixes the issue. It seems to only happen when i leave the PFsense web gui open in my browser for an extended period of time, like if i leave the Dashboard tab open overnight. The only package i have installed is openvpn-client-export. I hope this helps.

Edit: This is on version 2.3.2-RELEASE

Bryan Fehl wrote:

I just ran into this myself. Strangely, this issue causes all clients who try to connect with OpenVPN to just hang indefinitely.

That's normal, OpenVPN uses PHP scripts for authentication and some certificate verification. So if PHP is wedged, then OpenVPN can't authenticate.

Restarting Web Configurator & PHP fixes the issue. It seems to only happen when i leave the PFsense web gui open in my browser for an extended period of time, like if i leave the Dashboard tab open overnight. The only package i have installed is openvpn-client-export. I hope this helps.

Edit: This is on version 2.3.2-RELEASE

Which dashboard widgets do you have visible?

Jim Pingle wrote:

Which dashboard widgets do you have visible?

• System Information
• Picture
• Interfaces
• S.M.A.R.T. Status
• Gateways
• Thermal Sensors
• CARP Status
• NTP Status
• Services Status
• Traffic Graphs
• IPSec

I'm removing the IPsec widget based on recommendations I've seen in the forum where people had similar issues. Hopefully that prevents this from reoccurring.

Just Restarted PHP-FPM on a system with the following (no pfblocker installed):

• System Information
• Traffic Graphs
• Interfaces
• Gateways
• IPsec
• Interface Statistics

I've seen this symptom frequently with pfBlockerNG and large lists. I also don't run the IPsec widget.

The common thread I noticed is that there is a php process consuming 100% CPU for an extended length of time. It seems like there is some resource blocking somewhere.

What I've done that appears to help is increase the webUI process limit from 2 to 4. It's not perfect, but the instances of the webUI becoming totally unresponsive (and returning the 502 gateway error) have been fewer since making this change.

I can not even use it before it locks.

As soon as I try to change something, it loads a really long time before it locks.

Christoffer Öhman wrote:

I can not even use it before it locks.

As soon as I try to change something, it loads a really long time before it locks.

Do you have the IPSec widget on your dashboard? I removed that widget months ago and i haven't had this issue pop up since.

Bryan Fehl wrote:

Christoffer Öhman wrote:

I can not even use it before it locks.

As soon as I try to change something, it loads a really long time before it locks.

Do you have the IPSec widget on your dashboard? I removed that widget months ago and i haven't had this issue pop up since.

I'm sure I do not have IPSec up in the dashboard.

pfsense team:

Why is this bug being pushed back to another release yet again to a date that has not been determined? This issues causes an outage everyday. Lots of people are reporting this issue here and in the forums for over a year now.

I will be more than happy to volunteer my time to assist you to get to the bottom of this.

Please let me know how I can help.

Thanks for your offer. I have been working on this issue all week, sadly without getting very far because each diagnostic step takes so long.

What I am doing is cranking the polling frequency up to maximum (General setup page) and adding widgets one at a time. I'm trying to learn if this issue is caused by a particular widget, a combination of widgets, the polling frequency or ? I'm also watching the memory state, cpu load etc as I make each test.

If you have the ability to do that type of testing, or have any other ideas on the subject I would love to hear.

Thanks!

All I have to do to cause this is just leave the dashboard web page open. The problem happens anywhere from 1 hour to a day or so - across about 75 firewalls. I have started to close the web page to minimize the burden of the outages.

I have 3 columns with 5 second updates. The following widgets are in the dashboard for majority of the systems: System Information, Interfaces, Gateways, Interface Statistics, IPSec, NTP Status, Traffic Graph (1 sec updates)

Having a fair amount of experience myself managing php hosting systems I can offer some thoughts.

On my own pfsense unit I have seen this behaviour, when I was watching the system whilst it was happening I observed it was caused by background php scripts been busy perhaps tieing up the php-fpm server processes.

As an experiment I manually adjusted the php-fpm server configuration so there is more children running and the problem went away since.

It can be adjusted in /usr/local/etc/php-fpm.conf if anyone wants to experiment.

Given 2.4.1 wont support nano type systems anymore I expect memory usage can be loosened up a bit in terms of how restrictive things are configured to save resources.

Chris Collins wrote:

As an experiment I manually adjusted the php-fpm server configuration so there is more children running and the problem went away since.
It can be adjusted in /usr/local/etc/php-fpm.conf if anyone wants to experiment.
Given 2.4.1 wont support nano type systems anymore I expect memory usage can be loosened up a bit in terms of how restrictive things are configured to save resources.

The low number of processes/children apparently also is an issue with busy captive portals: https://forum.pfsense.org/index.php?topic=136847

So yeah, this should be relaxed by default since nano + i386 are gone, plus a GUI knob to have this configurable would be useful.

Running for 12+hours, dashboard up, IPSec widget (and many others including pfBlocker) loaded, no issues. (2.4.2.a.20171103.1355)
Update: been running for well over a day, still no problems.

Chris Collins wrote:

As an experiment I manually adjusted the php-fpm server configuration so there is more children running and the problem went away since.
It can be adjusted in /usr/local/etc/php-fpm.conf if anyone wants to experiment.
Given 2.4.1 wont support nano type systems anymore I expect memory usage can be loosened up a bit in terms of how restrictive things are configured to save resources.

I was having this problem and changing max children to 10 fixed my issue - Thank you kind sir!