pfSense

Replicating what I said in Slack: it'd be good to attach truss to one of the pfctl processes, to see what it's doing. A `procstat -k <pid>` to get the kernel stack would also be interesting.

Aside from that we should also consider introducing a new ioctl for this, because retrieving all states to just get the hostids (likely just two different ones!) is a very slow operation, and we could make that a lot faster by doing the loop in the kernel instead.
To be clear: there's absolutely a bug here. These processes should return, and we're going to try to fix that, but there's also an opportunity to let systems avoid a whole lot of pointless work.

Kristof Provost wrote in #note-2:

Replicating what I said in Slack: it'd be good to attach truss to one of the pfctl processes, to see what it's doing. A `procstat -k <pid>` to get the kernel stack would also be interesting.

Aside from that we should also consider introducing a new ioctl for this, because retrieving all states to just get the hostids (likely just two different ones!) is a very slow operation, and we could make that a lot faster by doing the loop in the kernel instead.
To be clear: there's absolutely a bug here. These processes should return, and we're going to try to fix that, but there's also an opportunity to let systems avoid a whole lot of pointless work.

Shell Output - procstat -k 45380

PID    TID COMM                TDNAME              KSTACK
45380 100666 pfctl               -                   <running>

So the lack of kernel stack as well as the lack of truss output (reported on Slack) would point in the direction of this being a userspace problem.
It's not immediately clear to me where we could be ending up in this loop in pfctl, but that's a useful clue at least.

I believe the problem is that we're overflowing the size field in the DIOCGETSTATESV2 call, and that's causing confusion between the kernel and userspace, which results in userspace looping forever.

I'm going to see if we can change the signed int to an unsigned one, which will buy us a bit of breathing room. Once that's done I'm also going to see about replacing this awful pfctl -ss | awk | sort construct with something that summarises the relevant information directly in the kernel.

I have a fix for the infinite pfctl loop, and in-progress patches for the improved code to retrieve creator ids. It ought to still be possible for 23.09.

I've merged the fix for the pfctl loop, as well as the new 'list creator ids' command.
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/1075 is still required to actually make use of that.

Kristof Provost wrote in #note-8:

I've merged the fix for the pfctl loop, as well as the new 'list creator ids' command.
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/1075 is still required to actually make use of that.

Running the latest development build of 23.09 I applied the diff here as a patch. The State Creator Host IDs field is broken and empty. Reverting the patch restored functionality. Are there any other patches needed to test?

You. do also need the kernel and pfctl changes. I'm not sure if there's been a successful build since those landed.
The easiest way to verify is to see if `pfctl -sc` produces the creator id list.

On a current snapshot the `pfctl -sc` changes are present and working on status_carp.php and the CLI. I pushed a small correction to ensure it's using the full path to tail in status_carp.php and re-added a sort to be sure the output is in the expected order. I didn't see anything in the pfctl code that sorted the output, but if it's there and I just missed it, then that sort command could come back out.

The kernel does not sort the list (and neither does pfctl). I had assumed that the sort was only there to ensure we had unique entries, so adding it again was the correct thing to do.