pfSense

I am also having this issue. Started with 21.02 (2.5)

Matt Johnson wrote:

https://github.com/MonkWho/pfatt

Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.

Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5

The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.

Hayden Hill wrote:

I am also having this issue. Started with 21.02 (2.5)

Matt Johnson wrote:

https://github.com/MonkWho/pfatt

Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.

Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5

The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.

I am as well...

I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique

Jordan Greene wrote:

I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique

Would you be willing to share your exact script? Are you using one of the pfatt scripts or something else using wpa_supplicant.

Jordan Greene wrote:

I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique

Please post your script, I am using x86 hardware (6700k, 16GB RAM, Intel 350t4 NIC, nvme) from an old gaming rig and have never had any issues until this latest upgrade.

Also running PfBlockerNG, suricata, ntopng, telegraf and the openvpn client export packages but have been running these for quite some time and never had issues until 2.5

Would love to see your script (scrubbed of course) and would be great if this is just a script issue but I have tried multiple different scripts and they all do this.

Still seems to be an issue on 2.5.1 release, if there is any extended logging or debug mode required to help troubleshoot this issue my system is available.

@Jordan Greene if you could post your script that is working I would love to give it a shot.

I am having the same problem - 2.5.0 and 2.5.1 both have 100% CPU load on one core. I have 4 cores assigned to the pfsense VM, so all this is wasting is electrical power and adding CO2 to the environment, but it is annoying.

Do folks need any data from me on config or debugging info?

Is this related to this bug in freebsd? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252844 It seems likely since we start it before the routing table is populated as well. Seems like the patch there would be applicable here.

MILO MEDIN wrote:

I am having the same problem - 2.5.0 and 2.5.1 both have 100% CPU load on one core. I have 4 cores assigned to the pfsense VM, so all this is wasting is electrical power and adding CO2 to the environment, but it is annoying.

Do folks need any data from me on config or debugging info?

Is this related to this bug in freebsd? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252844 It seems likely since we start it before the routing table is populated as well. Seems like the patch there would be applicable here.

Sorry I am not very well versed in code development, is there a way we can test this fix out on a live pfsense install?

@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521

Initial reports are good that it resolves the issue.

@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.

rom racer wrote:

@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521

Initial reports are good that it resolves the issue.

@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.

I can confirm this patch fixes the issue. I would love for this to become part the official build!

Hayden Hill wrote:

rom racer wrote:

@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521

Initial reports are good that it resolves the issue.

@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.

I can confirm this patch fixes the issue. I would love for this to become part the official build!

Also confirming this resolves the issue.

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

MILO MEDIN wrote:

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

Agreed, thank you so much @rom racer!

MILO MEDIN wrote:

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

I can confirm it fixes the issue as well.

Greg Revelle wrote:

MILO MEDIN wrote:

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

I can confirm it fixes the issue as well.

I have simplified steps to mitigate the issue here:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-830450022

Has this been been integrated to to the 2.6 development branch yet?

This is in 2.6 snapshots and now 2.5.2. Also in 21.09 snapshots if testing on arm.

@Renato please re-open this bug.

There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.

Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant

rom racer wrote:

@Renato please re-open this bug.

There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.

Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant

We don't need to care about /usr/sbin/wpa_supplicant because interfaces.inc code will always use `/usr/local/sbin/wpa_supplicant` unless port is not installed, what is never going to happen since it's a dependency of pfSense main port.

rom racer wrote:

@Renato please re-open this bug.

There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.

Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant

JIC I checked that that commit is present on 2.5.2 branch - https://github.com/pfsense/FreeBSD-src/commit/61c7d15d84f80ae1d92b42dc2da56ad94a80b46b#diff-7de31aa0a85ba44ddc629a7a17c7d356c5064f1335bccb09a8d98f86160c72e4

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

rom racer wrote:

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?

Hayden Hill wrote:

rom racer wrote:

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?

We do not support any external code that is added to pfSense, but in this case anyway, both wpa_supplicant binaries (base and ports) are patched.

Renato Botelho wrote:

Hayden Hill wrote:

rom racer wrote:

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?

We do not support any external code that is added to pfSense, but in this case anyway, both wpa_supplicant binaries (base and ports) are patched.

Understood and greatly appreciate it!

Jim Pingle wrote:

Due to changes in the freebsd-src branch used to build 2.5.2 snapshots, this needs re-tested on a build dated after this comment.

Looks like @Renato made the code change in the RELENG_2_5_0 source so hopefully this is still resolved! Thanks!

Why was this closed versus resolved?

Doesn't really matter, they're both closed states.

Tested under pfSense 2.5.2 released today and confirmed this is resolved. Thanks to everyone for helping get this done, and thanks to the pfSense team for including this fix.