Project

General

Profile

Actions

Bug #11453

closed

``wpa_supplicant`` uses 100% of a CPU core at boot

Added by Matt Johnson about 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Category:
Wireless
Target version:
Start date:
02/18/2021
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
22.01
Release Notes:
Default
Affected Version:
Affected Architecture:

Description

https://github.com/MonkWho/pfatt

Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.

Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5

The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.

Actions #1

Updated by Hayden Hill about 3 years ago

I am also having this issue. Started with 21.02 (2.5)

Matt Johnson wrote:

https://github.com/MonkWho/pfatt

Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.

Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5

The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.

Actions #2

Updated by Greg Revelle about 3 years ago

Hayden Hill wrote:

I am also having this issue. Started with 21.02 (2.5)

Matt Johnson wrote:

https://github.com/MonkWho/pfatt

Part of the project above is to use netgraph as a way to bypass the at&t provided gateway, netgraph is required to be able to set VLAN 0 on the 802.1x traffic to be accepted by the at&t network.

Something has changed in the new pfsense 2.5 release that causes the wpa_supplicant code to use up 100% cpu usage of a single core, I have tried using another script as well have others but this seems to be repeatable for everyone who upgrades to the latest RC's of 2.5

The last time I tried 2.5 was way back in summer of 2020 and this issue didn't exist at that time but I wasn't updating every day so I am not quite sure when this bug was introduced.

I am as well...

Actions #3

Updated by Jordan G about 3 years ago

I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique

Actions #4

Updated by Hayden Hill about 3 years ago

Jordan Greene wrote:

I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique

Would you be willing to share your exact script? Are you using one of the pfatt scripts or something else using wpa_supplicant.

Actions #5

Updated by Matt Johnson about 3 years ago

Jordan Greene wrote:

I'm using this currently as well but have not encountered any issues with CPU usage on 21.02 --- additional information is required regarding hardware in use and setup characteristics that may be unique

Please post your script, I am using x86 hardware (6700k, 16GB RAM, Intel 350t4 NIC, nvme) from an old gaming rig and have never had any issues until this latest upgrade.

Also running PfBlockerNG, suricata, ntopng, telegraf and the openvpn client export packages but have been running these for quite some time and never had issues until 2.5

Would love to see your script (scrubbed of course) and would be great if this is just a script issue but I have tried multiple different scripts and they all do this.

Actions #6

Updated by Matt Johnson almost 3 years ago

Still seems to be an issue on 2.5.1 release, if there is any extended logging or debug mode required to help troubleshoot this issue my system is available.

@Jordan Greene if you could post your script that is working I would love to give it a shot.

Actions #7

Updated by MILO MEDIN almost 3 years ago

I am having the same problem - 2.5.0 and 2.5.1 both have 100% CPU load on one core. I have 4 cores assigned to the pfsense VM, so all this is wasting is electrical power and adding CO2 to the environment, but it is annoying.

Do folks need any data from me on config or debugging info?

Is this related to this bug in freebsd? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252844 It seems likely since we start it before the routing table is populated as well. Seems like the patch there would be applicable here.

Actions #8

Updated by Matt Johnson almost 3 years ago

MILO MEDIN wrote:

I am having the same problem - 2.5.0 and 2.5.1 both have 100% CPU load on one core. I have 4 cores assigned to the pfsense VM, so all this is wasting is electrical power and adding CO2 to the environment, but it is annoying.

Do folks need any data from me on config or debugging info?

Is this related to this bug in freebsd? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252844 It seems likely since we start it before the routing table is populated as well. Seems like the patch there would be applicable here.

Sorry I am not very well versed in code development, is there a way we can test this fix out on a live pfsense install?

Actions #9

Updated by rom racer almost 3 years ago

@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521

Initial reports are good that it resolves the issue.

@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.

Actions #10

Updated by Hayden Hill almost 3 years ago

rom racer wrote:

@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521

Initial reports are good that it resolves the issue.

@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.

I can confirm this patch fixes the issue. I would love for this to become part the official build!

Actions #11

Updated by Matt Johnson almost 3 years ago

Hayden Hill wrote:

rom racer wrote:

@Milo Medin, great find! I've published some details on the pfatt issue here as well as a patched wpa_supplicant:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-826134521

Initial reports are good that it resolves the issue.

@pfSense team, what does it take to get the FreeBSD patch reviewed here (https://reviews.freebsd.org/D28249) backported to pfSense? I think that patch is only officially part of FreeBSD 13.0 so would not be part of FreeBSD 12.2 by default.

I can confirm this patch fixes the issue. I would love for this to become part the official build!

Also confirming this resolves the issue.

Actions #12

Updated by MILO MEDIN almost 3 years ago

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

Actions #13

Updated by Matt Johnson almost 3 years ago

MILO MEDIN wrote:

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

Agreed, thank you so much @rom racer!

Actions #14

Updated by Greg Revelle almost 3 years ago

MILO MEDIN wrote:

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

I can confirm it fixes the issue as well.

Actions #15

Updated by C HL almost 3 years ago

Greg Revelle wrote:

MILO MEDIN wrote:

@rom racer, thanks for doing the build.

I loaded it in 2.5.1 and can confirm it fixes the issue for me too.

I can confirm it fixes the issue as well.

I have simplified steps to mitigate the issue here:

https://github.com/MonkWho/pfatt/issues/41#issuecomment-830450022

Actions #16

Updated by MILO MEDIN almost 3 years ago

Has this been been integrated to to the 2.6 development branch yet?

Actions #17

Updated by Steve Wheeler almost 3 years ago

This is in 2.6 snapshots and now 2.5.2. Also in 21.09 snapshots if testing on arm.

Actions #18

Updated by Jim Pingle almost 3 years ago

  • Subject changed from wpa_supplicant cpu usage on boot to ``wpa_supplicant`` uses 100% of a CPU core at boot
  • Target version set to 2.5.2
  • Plus Target Version set to 21.09
Actions #19

Updated by Renato Botelho almost 3 years ago

  • Category changed from Unknown to Wireless
Actions #20

Updated by Renato Botelho almost 3 years ago

  • Status changed from New to Feedback
  • Assignee set to Renato Botelho
Actions #21

Updated by Renato Botelho almost 3 years ago

  • Status changed from Feedback to Resolved

This fix was committed on ports on wpa_supplicant version 2.9_3. We are now using 2.9_10.

Actions #22

Updated by rom racer almost 3 years ago

@Renato please re-open this bug.

There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.

Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant

Actions #23

Updated by Renato Botelho almost 3 years ago

rom racer wrote:

@Renato please re-open this bug.

There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.

Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant

We don't need to care about /usr/sbin/wpa_supplicant because interfaces.inc code will always use `/usr/local/sbin/wpa_supplicant` unless port is not installed, what is never going to happen since it's a dependency of pfSense main port.

Actions #24

Updated by Renato Botelho almost 3 years ago

rom racer wrote:

@Renato please re-open this bug.

There's two versions of wpa_supplicant included in pfSesnse. Both the version in ports (/usr/local) and the "system" version (sorry I don't know what you call that) in /usr.

Please confirm you have fixed both instances of wpa_supplicant. Your most recent comment only confirms ports, which is only half the problem. Commit history here seems to confirm you have not fixed this version: https://github.com/pfsense/FreeBSD-src/tree/devel-12/usr.sbin/wpa/wpa_supplicant

JIC I checked that that commit is present on 2.5.2 branch - https://github.com/pfsense/FreeBSD-src/commit/61c7d15d84f80ae1d92b42dc2da56ad94a80b46b#diff-7de31aa0a85ba44ddc629a7a17c7d356c5064f1335bccb09a8d98f86160c72e4

Actions #25

Updated by rom racer almost 3 years ago

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

Actions #26

Updated by Hayden Hill almost 3 years ago

rom racer wrote:

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?

Actions #27

Updated by Renato Botelho almost 3 years ago

Hayden Hill wrote:

rom racer wrote:

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?

We do not support any external code that is added to pfSense, but in this case anyway, both wpa_supplicant binaries (base and ports) are patched.

Actions #28

Updated by Hayden Hill almost 3 years ago

Renato Botelho wrote:

Hayden Hill wrote:

rom racer wrote:

I don't know what interfaces.inc is but if you read the original description of this bug, this was encountered in an open source project that runs on top of pfSense. That open source project uses "/usr/sbin/wpa_supplicant" which is (currently) broken. If "/usr/sbin/wpa_supplicant" shouldn't be used, it should be removed. It doesn't make sense to include binary executables that aren't used.

Thanks for checking for the commit; I looked in the wrong path in a hurry.

@Renato. This specific script we all are using that brought this issue to light runs at start-up and calls for /usr/sbin/wpa_supplicant. Is it possible to get that patched as well?

We do not support any external code that is added to pfSense, but in this case anyway, both wpa_supplicant binaries (base and ports) are patched.

Understood and greatly appreciate it!

Actions #29

Updated by Jim Pingle over 2 years ago

  • Status changed from Resolved to Feedback

Due to changes in the freebsd-src branch used to build 2.5.2 snapshots, this needs re-tested on a build dated after this comment.

Actions #30

Updated by Hayden Hill over 2 years ago

Jim Pingle wrote:

Due to changes in the freebsd-src branch used to build 2.5.2 snapshots, this needs re-tested on a build dated after this comment.

Looks like @Renato made the code change in the RELENG_2_5_0 source so hopefully this is still resolved! Thanks!

Actions #31

Updated by Jim Pingle over 2 years ago

  • Status changed from Feedback to Closed
Actions #32

Updated by Hayden Hill over 2 years ago

Why was this closed versus resolved?

Actions #33

Updated by Jim Pingle over 2 years ago

Doesn't really matter, they're both closed states.

Actions #34

Updated by rom racer over 2 years ago

Tested under pfSense 2.5.2 released today and confirmed this is resolved. Thanks to everyone for helping get this done, and thanks to the pfSense team for including this fix.

Actions #35

Updated by Jim Pingle over 2 years ago

  • Plus Target Version changed from 21.09 to 22.01
Actions

Also available in: Atom PDF