pfSense » pfSense Plus

could be related: #11436 #11418

If you can re-enable those and test again, monitor the CPU usage, CPU temp, and so on to see if they are unusually high before/during the crashes.

Also, don't attach them here but look on the disk for core files and keep copies of them (e.g. find / -name "*.core"), they may or may not be present depending on the state of the system at the time of the crash.

There is a fix that passes my testing here:

https://reviews.freebsd.org/D28821

The above patch is for FreeBSD HEAD but applies to 12.2-STABLE for pfSense. We're working on a merge and patch release now

Scott Lang, that tracks along the same lines with the issues I was having back in Sep 2020: https://forum.netgate.com/topic/155919/debug-kernel-build-with-witness

I was noticing a kernel memory leak, that was overwriting the priority tracker for the pf_rules_lock, which caused the kernel to enter a deadlock, since it can't release the mutex (rmlock).

After the Problem occurred first time I applied the quick fix setting to 1 CPU in the loader.conf > hw.ncpu=1

Now I applied 21.02.p1 to SG-3100 yesterday Evening, after the reboot of the Updated I removed the before manually added hw.ncpu=1 from the loader.conf. It didn't take 30 min to bring the Problem back on 21.02.p1 not passing traffic anymore.

Question: Was 21.02.p1 just a quick fix addind a cpu limit to laoder.conf or was the membar already applied? First would explain why after removing hw.ncpu=1 the system got the problem again.

same issue after upgrading to 21.02-p1:

pid 833 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

but if I disable pfBlockerNG (3.0.0_10) and reboot it works again,

If I enable pfBlockerNG (python mode) it stops passing traffic and I see again:

pid 357 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)

What Viktor mentioned could be a reason. In my tested and still failing SG-3100 it also used the pfBlockerNG-dev package.
I will have a look into it and test in different scenarios if this is the reason for the pf lockup.

Marco Goetze wrote:

Question: Was 21.02.p1 just a quick fix addind a cpu limit to laoder.conf or was the membar already applied? First would explain why after removing hw.ncpu=1 the system got the problem again.

No. It was a real fix in the kernel:

• https://reviews.freebsd.org/D28821
• https://www.netgate.com/blog/pfsense-obscure-bugs-and-code-wizards.html

You may be seeing a different issue or a variation.

Viktor Gurov wrote:

but if I disable pfBlockerNG (3.0.0_10) and reboot it works again,

If I enable pfBlockerNG (python mode) it stops passing traffic and I see again:
[...]

I don't think that's the same issue, but it bears investigating. I'd start a new issue with the details of that so it doesn't get mixed up with this. If research shows it's the same or similar, then the redundant one can be closed.

Let me share some of mny observartions in the last 3 days.

• hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
• hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
• hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15h

When the lockup happens I can only use the Serial Console to reach the FW, if I issue "pfctl -d" afterwards access is restored.
Interessting thing is my syslog seems messed up there are a few entries with totally wrong timestamps maybe NTP not synced. But no Signal 11 in the systemlog even the lock occurs.

For mow I keep all ext. packages disabled.