pfSense

This is weird, but if configuring Limiters with CoDel AQM and QFQ Scheduler, it works. Problems exists with default AQM Taildrop and default scheduler FIFO.

Victor Preatoni wrote:

This bug may be related to #8956
But it's a different situation...

To get around bug #8956 I just manually deleted all limiter information in XML config file and started from scratch.
After that, queues can be created properly under each limiter.
See https://forum.netgate.com/assets/uploads/files/1537917051995-1.png

But, after assigning traffic to In/Out pipes, dynamic queues are empty. Tried resetting states, rebooting firewall, but no traffic into the Limiter queue:
[...]

Problem started after upgrading from 2.4.3 to 2.4.4

Far more likely is that it is working properly but just not showing the traffic in the queues in the diagnostic output with certain schedulers.

Has anyone ran tests to see if things are being shaped properly in these cases, without looking at Diag > Limiter info?

Tried to set a very hard limit on my DownloadLimiter and seems to be shaping properly. Tested with testmy.net

using limiters with queues works fine with codel and fq_codel its just that we r not able to see it in limiter info and a constant spam of config_aqm Unable to configure flowset, flowset busy!

using limiters without child queues also works and able to see in limiter info but again the constant spam of above message

Seeing same as all the aforementioned comments. Taildrop and FIFO do work, but don't show under Diag > Limiter Info. Switch to Codel and QFQ, these also work, and now also show under Diag > Limiter Info.

Had to switch back to Taildrop/FIFO, though the limiters are no longer possible to monitor.

With QFQ, getting sudden flood of these and then the whole system crashes:

Oct 9 14:31:03 kernel qfq_dequeue BUG/* non-workconserving leaf */

Can see that our traffic shaper is nonfunctional now as of 2.4.4 in terms of per-host dynamic bandwidth shaping.

Work around for the missing queues/inability to create queues in 2.4.4 was to delete all limiters & queues, then recreate them, then reassign queues to firewall rules. Then, found out that the limiter diagnostic info was not functional with Taildrop/FIFO. Codel/QFQ caused system to crash eventually.

Whole point of the setup was to do the amazing per-host dynamic bandwidth dividing that pfsense was so good with. Can confirm now that although the limiters/queues are recreated and working to limit the maximum aggregate bandwidth, the mask by destination (Down_LAN) or sources addresses (Up_LAN) does not seem to work. These queues are under the DownLimiter and UpLimiter limiters. Up_LAN is assigned to In pipe on a LAN interface firewall rule and Down_LAN is assigned to Out pipe. All was working before 2.4.4! The hosts always showed identical traffic during peak usage, dividing the total bandwidth evenly. This is nonfunctional now.

Any way I can directly verify that traffic is actually going through per-host queues?

Samir Patel wrote:

Had to switch back to Taildrop/FIFO, though the limiters are no longer possible to monitor.

Qith QFQ, getting sudden flood of these and then the whole system crashes:

Oct 9 14:31:03 kernel qfq_dequeue BUG/* non-workconserving leaf */

I got that issue a few times too, syslog flooded, and then I had to manually reboot as pfSense crashed completely.

Victor Preatoni wrote:

I got that issue a few times too, syslog flooded, and then I had to manually reboot as pfSense crashed completely.

With further testing, I don't see any evidence that Taildrop/FIFO works to actually have the traffic go through child queues. Codel/QFQ does, but that eventually crashes the system. Finally, Codel/Round-robin seemed to actually work (as it used to with Taildrop/FIFO). Traffic goes to child queues and the mask by source/destination addresses functions as expected to dynamically shape bandwidth per-host. Would be nice if Taildrop/FIFO work and then we can compare with the performance of Codel/Round-robin.

A quick data point to confirm what Victor and Samir observed:

• I run two pfsense boxes at different locations. The key feature we rely on is using limiters to shape traffic to handle small-bandwidth internet connections and lots of users.
• In both deployments, due to #8956, I had to re-create our limiter/queues after upgrading to 2.4.4.
• Due to this bug, I had to set the configuration of the limiters to use Codel/QFQ and the queues to Codel for shaping to work. I verified with a basic test network that queues are not working with TailDrop/FIFO.
• Every 10-24 hours, both boxes would crash after producing a bunch of errors along the lines of...
  

     kernel qfq_dequeue BUG/* non-workconserving leaf */
     

• After trying to use to Codel / PRIO on the limiters, to avoid the crashes, I ended up in a crash-loop and had to boot to a recovery disk, and manually update the `conf.xml` to stop using PRIO.

At this point, I've just disabled the limiters / queues. It's better for people to deal with the problems caused by no traffic shaping than have no network access :-).

FWIW, I really think this should be addressed in the same release as #8956. Unless you can both define queues and use them, the feature is still effectively broken.

Terence Kent wrote:

At this point, I've just disabled the limiters / queues. It's better for people to deal with the problems caused by no traffic shaping than have no network access :-).

Try Codel/Round-Robin. This seems to work and has been stable a couple of days now.

Samir Patel wrote:

Terence Kent wrote:

At this point, I've just disabled the limiters / queues. It's better for people to deal with the problems caused by no traffic shaping than have no network access :-).

Try Codel/Round-Robin. This seems to work and has been stable a couple of days now.

I'm trying it now. Let's see how it goes.

Big shame this bug report has not been escalated to Critical Priority. Server having kernel panics, hanging or rebooting themselves is SERIOUS ISSUE.

Samir Patel wrote
...Try Codel/Round-Robin. This seems to work and has been stable a couple of days now.

Thanks! I'm trying that combination at the less-critical location now, I'll update here if I see crashes again.

Victor Preatoni wrote
...Big shame this bug report has not been escalated to Critical Priority. Server having kernel panics, hanging or rebooting themselves is SERIOUS ISSUE.

I agree. This is one of those bugs that is worse than it seems at first blush. Since the kernel panic loop I encountered was caused by the pfsense configuration I selected in the UI, that means HA setups that have replicated configuration can be be taken down. When HA setups go down, there are lots of uncomfortable meetings afterwards.

Thanks Luiz and Jim!

While on 2.4.4, I manually switched to Worst-case Weighted fair Queueing (WF2Q+) and seems to be working fine.

I just noticed the updates - thanks for the fix and explanation Luiz!