Bug #11453
closed``wpa_supplicant`` uses 100% of a CPU core at boot
Added by Matt Johnson over 3 years ago. Updated almost 3 years ago.
0%
Description
https://github.com/MonkWho/pfatt
Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.
Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5
The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.
Updated by Hayden Hill over 3 years ago
I am also having this issue. Started with 21.02 (2.5)
Matt Johnson wrote:
https://github.com/MonkWho/pfatt
Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.
Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5
The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.
Updated by Greg Revelle over 3 years ago
Hayden Hill wrote:
I am also having this issue. Started with 21.02 (2.5)
Matt Johnson wrote:
https://github.com/MonkWho/pfatt
Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.
Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5
The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.
I am as well...
Updated by Jordan G over 3 years ago
I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique
Updated by Hayden Hill over 3 years ago
Jordan Greene wrote:
I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique
Would you be willing to share your exact script? Are you using one of the pfatt scripts or something else using wpa_supplicant.
Updated by Matt Johnson over 3 years ago
Jordan Greene wrote:
I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique
Please post your script, I am using x86 hardware (6700k, 16GB RAM, Intel 350t4 NIC, nvme) from an old gaming rig and have never had any issues until this latest upgrade.
Also running PfBlockerNG, suricata, ntopng, telegraf and the openvpn client export packages but have been running these for quite some time and never had issues until 2.5
Would love to see your script (scrubbed of course) and would be great if this is just a script issue but I have tried multiple different scripts and they all do this.
Updated by Matt Johnson over 3 years ago
Still seems to be an issue on 2.5.1 release, if there is any extended logging or debug mode required to help troubleshoot this issue my system is available.
@Jordan Greene if you could post your script that is working I would love to give it a shot.
Updated by MILO MEDIN over 3 years ago
I am having the same problem - 2.5.0 and 2.5.1 both have 100% CPU load on one core. I have 4 cores assigned to the pfsense VM, so all this is wasting is electrical power and adding CO2 to the environment, but it is annoying.
Do folks need any data from me on config or debugging info?
Is this related to this bug in freebsd? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252844 It seems likely since we start it before the routing table is populated as well. Seems like the patch there would be applicable here.
Updated by Matt Johnson over 3 years ago
MILO MEDIN wrote:
I am having the same problem - 2.5.0 and 2.5.1 both have 100% CPU load on one core. I have 4 cores assigned to the pfsense VM, so all this is wasting is electrical power and adding CO2 to the environment, but it is annoying.
Do folks need any data from me on config or debugging info?
Is this related to this bug in freebsd? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252844 It seems likely since we start it before the routing table is populated as well. Seems like the patch there would be applicable here.
Sorry I am not very well versed in code development, is there a way we can test this fix out on a live pfsense install?
Updated by rom racer over 3 years ago
@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:
https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521
Initial reports are good that it resolves the issue.
@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.
Updated by Hayden Hill over 3 years ago
rom racer wrote:
@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:
https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521
Initial reports are good that it resolves the issue.
@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.
I can confirm this patch fixes the issue. I would love for this to become part the official build!
Updated by Matt Johnson over 3 years ago
Hayden Hill wrote:
rom racer wrote:
@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:
https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521
Initial reports are good that it resolves the issue.
@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.
I can confirm this patch fixes the issue. I would love for this to become part the official build!
Also confirming this resolves the issue.
Updated by MILO MEDIN over 3 years ago
@rom racer, thanks for doing the build.
I loaded it in 2.5.1 and can confirm it fixes the issue for me too.
Updated by Matt Johnson over 3 years ago
MILO MEDIN wrote:
@rom racer, thanks for doing the build.
I loaded it in 2.5.1 and can confirm it fixes the issue for me too.
Agreed, thank you so much @rom racer!
Updated by Greg Revelle over 3 years ago
MILO MEDIN wrote:
@rom racer, thanks for doing the build.
I loaded it in 2.5.1 and can confirm it fixes the issue for me too.
I can confirm it fixes the issue as well.
Updated by C HL over 3 years ago
Greg Revelle wrote:
MILO MEDIN wrote:
@rom racer, thanks for doing the build.
I loaded it in 2.5.1 and can confirm it fixes the issue for me too.
I can confirm it fixes the issue as well.
I have simplified steps to mitigate the issue here:
https://github.com/MonkWho/pfatt/issues/41#issuecomment-830450022
Updated by MILO MEDIN over 3 years ago
Has this been been integrated to to the 2.6 development branch yet?
Updated by Steve Wheeler about 3 years ago
This is in 2.6 snapshots and now 2.5.2. Also in 21.09 snapshots if testing on arm.
Updated by Jim Pingle about 3 years ago
- Subject changed from wpa_supplicant cpu usage on boot to ``wpa_supplicant`` uses 100% of a CPU core at boot
- Target version set to 2.5.2
- Plus Target Version set to 21.09
Updated by Renato Botelho about 3 years ago
- Category changed from Unknown to Wireless
Updated by Renato Botelho about 3 years ago
- Status changed from New to Feedback
- Assignee set to Renato Botelho
Updated by Renato Botelho about 3 years ago
- Status changed from Feedback to Resolved
This fix was committed on ports on wpa_supplicant version 2.9_3. We are now using 2.9_10.
Updated by rom racer about 3 years ago
@Renato please re-open this bug.
There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.
Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant
Updated by Renato Botelho about 3 years ago
rom racer wrote:
@Renato please re-open this bug.
There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.
Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant
We don't need to care about /usr/sbin/wpa_supplicant because interfaces.inc code will always use `/usr/local/sbin/wpa_supplicant` unless port is not installed, what is never going to happen since it's a dependency of pfSense main port.
Updated by Renato Botelho about 3 years ago
rom racer wrote:
@Renato please re-open this bug.
There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.
Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant
JIC I checked that that commit is present on 2.5.2 branch - https://github.com/pfsense/FreeBSD-src/commit/61c7d15d84f80ae1d92b42dc2da56ad94a80b46b#diff-7de31aa0a85ba44ddc629a7a17c7d356c5064f1335bccb09a8d98f86160c72e4
Updated by rom racer about 3 years ago
I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.
Thanks for checking for the commit; I looked in the wrong path in a hurry.
Updated by Hayden Hill about 3 years ago
rom racer wrote:
I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.
Thanks for checking for the commit; I looked in the wrong path in a hurry.
@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?
Updated by Renato Botelho about 3 years ago
Hayden Hill wrote:
rom racer wrote:
I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.
Thanks for checking for the commit; I looked in the wrong path in a hurry.
@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?
We do not support any external code that is added to pfSense, but in this case anyway, both wpa_supplicant binaries (base and ports) are patched.
Updated by Hayden Hill about 3 years ago
Renato Botelho wrote:
Hayden Hill wrote:
rom racer wrote:
I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.
Thanks for checking for the commit; I looked in the wrong path in a hurry.
@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?
We do not support any external code that is added to pfSense, but in this case anyway, both wpa_supplicant binaries (base and ports) are patched.
Understood and greatly appreciate it!
Updated by Jim Pingle about 3 years ago
- Status changed from Resolved to Feedback
Due to changes in the freebsd-src branch used to build 2.5.2 snapshots, this needs re-tested on a build dated after this comment.
Updated by Hayden Hill about 3 years ago
Jim Pingle wrote:
Due to changes in the freebsd-src branch used to build 2.5.2 snapshots, this needs re-tested on a build dated after this comment.
Looks like @Renato made the code change in the RELENG_2_5_0 source so hopefully this is still resolved! Thanks!
Updated by Jim Pingle about 3 years ago
Doesn't really matter, they're both closed states.
Updated by rom racer about 3 years ago
Tested under pfSense 2.5.2 released today and confirmed this is resolved. Thanks to everyone for helping get this done, and thanks to the pfSense team for including this fix.
Updated by Jim Pingle almost 3 years ago
- Plus Target Version changed from 21.09 to 22.01