pfSense

When a dummynet pipe with a delay is applied to traffic on a secondary WAN, reply traffic is dropped. It seems that the fix in #15220 does not take effect in this scenario.

Test setup:
vmx1 is WAN1, vmx2 is WAN2

# match rule -- pfctl -vvsr
@296 match in on vmx2 inet all label "USER_RULE: QoS queue default (outside) IPv4" label "id:1686509600" ridentifier 1686509600 dnqueue(12, 9) ! tagged blocklist
  [ Evaluations: 151142    Packets: 284       Bytes: 78078       States: 0     ]
  [ Inserted: uid 0 pid 0 State Creations: 0     ]
  [ Last Active Time: N/A ]

# pass rule -- pfctl -vvsr
@799 pass in quick on vmx2 reply-to (vmx2 192.168.1.254) inet proto udp from any to 127.0.0.1 port = rsf-1 keep state (if-bound) label "USER_RULE: OpenVPN" label "id:1679170153" ridentifier 1679170153 ! tagged blocklist
  [ Evaluations: 438       Packets: 301       Bytes: 106093      States: 0     ]
  [ Inserted: uid 0 pid 0 State Creations: 2     ]
  [ Last Active Time: Tue Mar 26 12:24:48 2024 ]

The following works: limiter queue without a delay on the pipe:

# pipe without delay -- dnctl pipe show
00004:  80.000 Mbit/s    0 ms burst 0
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active

# state info -- pfctl -vvss
vmx2 udp 127.0.0.1:1195 (192.168.1.253:1195) <- 172.58.109.152:61712       MULTIPLE:MULTIPLE
   age 00:00:11, expires in 00:00:51, 10:8 pkts, 3632:3280 bytes, rule 799
   id: dd6b0a6600000000 creatorid: af6c8b55 reply-to: 192.168.1.254@vmx2
   origif: vmx1

The following does not work: limiter queue with a 1ms delay on the pipe:

# pipe with 1ms delay -- dnctl pipe show
00004:  80.000 Mbit/s    1 ms burst 0
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active

# state info -- pfctl -vvss
all udp 127.0.0.1:1195 (192.168.1.253:1195) <- 172.58.109.152:64462       NO_TRAFFIC:SINGLE
   age 00:00:40, expires in 00:00:20, 5:0 pkts, 410:0 bytes, rule 799
   id: 7fe5096600000000 creatorid: af6c8b55 reply-to: 192.168.1.254@vmx2
   origif: vmx2