pfSense

Forget to mention: I am using VMWare workstation 15.5, 2 core PFsense VM with em adapters.

Update:

I've tested different pipe bandwidth and same 50 mbit traffic:

85 Mbit pipe - less reorder
185 Mbit pipe - more reorder
600 Mbit pipe - less reorder
1000 Mbit pipe - no reorder

When 85 Mbit pipe is saturated with traffic there is no reorder.

No, I did not test it on 2.5 so far.

Can you try reproduce this problem and fix it in release version?

I have tested 2.4.2, 2.4.5p1, 2.5 - all versions have this problem.

Setting kernel.hz=1000 instead of 100 does not fix it too.

Packet reordering makes PFSense shaping unusable because it degrades TCP performance from 85 Mbit to 20-30, and produce other errors.

If TCP session get full speed, then all working good. But if speed drops for some reason, TCP can not restore it due to out-of-order packets.

Adding 10 ms delay to the pipe seems to fix reordering.

Trying to set both kernel.hz=1000 and delay=1 ms to make a workaround lead to crashes under load.

Observed the same on 2.4.5 p1 with out of order packets during iperf testing using fq-codel with limiters set to 930 Mb/s. Tested it on a Supermicro X10SDV-4C-TLN2F without any packages installed and wasn't seeing any particular high load on it either.

Since net.inet.ip.dummynet.io_fast does split path of packets for saturated/unsaturated pipe mode, then this setting is likely to be responsible for packet reordering. (traffic is very bursty for TCP without pacing or IPERF3 UDP test, so saturation/desaturation of pipe occurs several times in one second, so it seems then we get reorders on every transition)

But setting of net.inet.ip.dummynet.io_fast=0 has no effect, net.inet.ip.dummynet.io_pkt_fast is still increasing. Explanation is very simple:
io_fast check is commented in dummynet source code:

if (/*dn_cfg.io_fast &&*/ m *m0 && (dir & PROTO_LAYER2) 0 ) {

https://github.com/luigirizzo/dummynet/blob/master/sys/netinet/ipfw/ip_dn_io.c

And the same commented code in pfsense repository.

https://github.com/pfsense/FreeBSD-src/blob/devel-12/sys/netpfil/ipfw/ip_dn_io.c

I have tried to disable whole if (/*dn_cfg.io_fast */ && ...) via patching /boot/kernel/dummynet.ko .

Traffic then goes only to net.inet.ip.dummynet.io_pkt, net.inet.ip.dummynet.io_pkt_fast always stays zero.

But whole pfsense hangs after several seconds.

There is a comment before this if block:
"XXX Don't call dummynet_send() if scheduler return the packet just enqueued. This avoid a lock order reversal."

This seems to be a cause of hang, and it is unclear how to turn off io_fast correctly.

I mentioned in previous post, adding 10 ms lag to pipe seems to solve the problem. This can be be explained this way: if lag is set, then it
actually disables io_fast, because dummynet redirect all packets to delay queue and does not perform fast io, so no reorder occurs.

Now I need an advice.

I've succesfully used kernel.hz=1000 and limiter delay=1ms as workaround to fix this problem.

I've also posted message to freebsd forum: https://forums.freebsd.org/threads/possible-race-condition-bug-in-dummynet-out-of-order-packets.78312/

So far I leave this problem for pfSense team, FreeBSD community, or anyone who want to help create proper fix. As I can see, dummynet is not maintained for a long time by the authors.