pfSense

A few additions:
- it seems to happen more often if pfSense is installed and used in a virtual environement
- it seems to happen more often if aliases are used in the firewall rules
- it seems to happen more often if a large ruleset is used, for example if pfBlockerNG is used and/or IPv6 with bogons block
- it seems to happen rarely if pfSense is installed in a single core environment

I can reproduce this at will. My hardware is a Supermicro 5018D-FN4T (Same as XG-1541). I can provide a config file if you have matching hardware in your test lab. I will add that the larger the total number of items in tables the more obvious the issue is but the issue is present (look at pfctl cpu usage compared to 2.4.4-p3) even when below the 65k hardlimit removed by #10254. Try a config with 250,000 or more total items and you can't miss the pfctl cpu spike and the resulting latency and packet loss when reloading tables.

I can confirm this problem. If there are changes to the routing table (because there is packet loss on some OpenVPN Interface or other) these firewall table reloads are triggered.

I have already attached pictures on this topic in my post - https://forum.netgate.com/topic/151921/pfsense-2-4-5-hohe-last-ipv6

The problem concerns 2 pfSense installations at my site and also the pfSense in our company is affected. On all of them IPv6 is active.

Is it possible that the IPv6 bogon list is simply too large to be processed on the pfSense?

+1 exactly the same issue here.

I've had to revert back to 2.4.4-p3 because the workaround doesn't work if you need to keep using pfBlockerNG. There aren't enough Firewall table entries to operate normally.

I am having the same issue here running pfsense on Proxmox. Enabling pfBlocker makes the network unusable as the CPU goes to 100% (16 cores on a DL380) and the gateway ping time approached 4000ms.

Not much to add, but getting same issue.

Not virtual - SG3100.
IPV6 enabled.
Snort + Pfblocker enabled.
Bogan block on.
Handful of manual rules - most are pfblocker country blocks.

During CPU spikes I can't ssh or use web interface. DNS down, however I can ping.

Also get "/tmp/rules.debug:23: cannot define table bogonsv6: Cannot allocate memory" error too.

+1
in my case it is the Filter Reload. I had this high CPU load every 15 minutes. All cores go to 100% for seconds. No packets lost but 4000-5000ms delay. VoIPs and VPNs were crashed.

So i found that the cron job "/etc/rc.filter_configure_sync" was the cause.
My workaround: I changed this job to midnight and rule changes are only out site of business hours. Now the 60 Ipsec-tunnls where stable again. But this is not the solution

Environment:
pfblocker with GeoIP selections
snort
FRR
CARP
Traffic Shaper
IPv6 disabled

Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
20 CPUs: 1 package(s) x 10 core(s) x 2 hardware threads

So I have the same issue on a Netgate SG3100. It starts when you enable multiple GeoIP regions on pfblockerng for my firewall. Once you start seeing that "cannot allocate memory" error no matter what max tables settings you do you will start seeing weird browser slow downs and potentially other symptoms. Even if you remove regions and reload pfblockerng it will still give you the error and symptoms.
What I had to do is disable all GeoIP regions with the exception of the items (PRI) that get enabled using the wizard, I also reverted any states or max table changes to the defaults and then perform a reboot of firewall. Once it reboots you should not see that error. you can then enable maybe top spammers and that is it. If you try to enable more regions etc well you most likely will start seeing the problems and errors. Reverting back to 2.4.4 was not an option for me since I had not made a backup before the 2.4.5 update. Hopefully this helps someone else that is having the same issues and is using pfblockerng.

For people suffering from this now, until the next release, this might help:
add the line below to /boot/loader.conf.local and reboot

kern.smp.disabled=1

Thx Luiz! this is the commit, right?
https://github.com/pfsense/FreeBSD-src/commit/6c7a5a8e69762db2ac0bc465f37c8f04a45a34ea#diff-cfe1b9af5640485f16600f6f505ad7a5

Just wondering, is this just a straight rollback of the original FreeBSD PR#230619? It only applies to the FreeBSD 11/2.4.5 branches right, not 12/2.5.0? Reading the description of the original commit, seems like it was an important race condition fix. Or do I mis-understand?

Luke,

From what we can tell, pf is doing a ton of smp rendezvous zeroing per-CPU counters. The described "hang" seems (mostly) caused by a ton of back-to-back IPIs. As you noted, disabling SMP makes the issue go away.

All of that said, the issue seems to be present upstream, and it's likely deeper than just #230619.

A proposed fix: https://reviews.freebsd.org/D24803

Imported markj@ fix to 2.5.0

https://svnweb.freebsd.org/base?view=revision&revision=360903

Luke Hamburg wrote:

Thx Luiz! this is the commit, right?
https://github.com/pfsense/FreeBSD-src/commit/6c7a5a8e69762db2ac0bc465f37c8f04a45a34ea#diff-cfe1b9af5640485f16600f6f505ad7a5

Just wondering, is this just a straight rollback of the original FreeBSD PR#230619? It only applies to the FreeBSD 11/2.4.5 branches right, not 12/2.5.0? Reading the description of the original commit, seems like it was an important race condition fix. Or do I mis-understand?

Luke,

That is correct, but the race described in commit only affect the counter numbers (maybe a crash if you aren't lucky) but seems more difficult to happen than the current issue, the tradeoff seems to be worth at this moment. Also, this is a temporary fix while the real fix is being developed and tested.

Testing a kernel with the original fix taken out (so r345177 restored), and the new fix applied, it still look good to me.

Same Hyper-V VM as before. No console lock, table loads fast, no CPU spike or persistent pfctl process.

: time pfctl -t bogonsv6 -T flush
111171 addresses deleted.
0.000u 0.066s 0:00.06 100.0%    373+192k 0+0io 0pf+0w
: time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
111171/111171 addresses added.
0.172u 0.267s 0:00.44 97.7%    364+187k 0+0io 0pf+0w
: time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
0/111171 addresses added.
0.190u 0.087s 0:00.27 100.0%    362+186k 0+0io 0pf+0w

We test more once we have RC snapshots.

I was able to replicate this to a lesser extent on Proxmox VE (6.2-4) as well, with a 4-core VM. Similar setup to the Hyper-V test before. It wasn't as prominent here like on Hyper-V but it was still very noticeable.

Stock 2.4.5-RELEASE kernel:

: time pfctl -t bogonsv6 -T flush
112536 addresses deleted.
0.000u 0.101s 0:00.10 100.0%    364+187k 0+0io 0pf+0w
: time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
112536/112536 addresses added.
0.259u 6.190s 0:06.45 99.8%    356+183k 0+0io 0pf+0w
: time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
0/112536 addresses added.
0.239u 0.147s 0:00.38 97.3%    381+198k 0+0io 0pf+0w

Notice it took >6 seconds of real time here. Console wasn't unresponsive but the network did take a hit and ssh was lagging.

Kernel with the later fix applied:

: time pfctl -t bogonsv6 -T flush
112536 addresses deleted.
0.000u 0.073s 0:00.07 100.0%    360+185k 0+0io 0pf+0w
: time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
112536/112536 addresses added.
0.251u 0.322s 0:00.57 100.0%    358+184k 0+0io 0pf+0w
: time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
0/112536 addresses added.
0.252u 0.134s 0:00.38 100.0%    361+185k 0+0io 0pf+0w