Regression #14039
closed
Limiters have no effect on upload traffic passed by policy routing rules
0%
Description
Upload traffic is not limited if the rule passing the traffic uses route-to. This last worked in pfSense+ 22.01 and pfSense CE 2.6.0. See https://redmine.pfsense.org/issues/13026#note-15:
Essentially what happens is that we have two states:
all tcp 10.0.2.1:5201 <- 192.168.1.100:44607 ESTABLISHED:ESTABLISHED [2078351244 + 3221291264] wscale 6 [1276678361 + 2419064832] wscale 6 age 00:00:03, expires in 24:00:00, 122313:60993 pkts, 183465201:3171644 bytes, rule 81 id: a5627f6300000000 creatorid: a17f7a2e gateway: 1.0.2.1 origif: vtnet2 all tcp 1.0.2.78:50878 (192.168.1.100:44607) -> 10.0.2.1:5201 ESTABLISHED:ESTABLISHED [1276678361 + 2419064832] wscale 6 [2078351244 + 3221291264] wscale 6 age 00:00:03, expires in 24:00:00, 122313:60993 pkts, 183465201:3171644 bytes, rule 77 id: a6627f6300000000 creatorid: a17f7a2e gateway: 1.0.2.1 origif: vtnet0
The first state is created by the rule with the limiter, but because that rule also does route-to the packet is passed through pf_test() a second time, which creates the second state. That second state is created by a rule which doesn't have the limiter associated, and that means that when it matches the limiter is not applied. It's that second state that ends up matching incoming packets, so the limiter doesn't get applied there.
Files
Updated by Marcos M about 1 year ago
The issue can be avoided by creating a floating rule that applies the upload limiter.
Without the floating rule, the download speed is limited and the upload speed is unlimited:
all icmp 1.0.0.1:1 <- 10.0.5.50:1 0:0 age 00:00:07, expires in 00:00:09, 7:7 pkts, 420:420 bytes, rule 705 id: 2734f26400000000 creatorid: 4da82510 gateway: 192.168.100.1 origif: vmx0.5 all icmp 192.168.100.11:36640 (10.0.5.50:1) -> 1.0.0.1:36640 0:0 age 00:00:07, expires in 00:00:09, 7:7 pkts, 420:420 bytes, rule 155 id: 2834f26400000000 creatorid: 4da82510 gateway: 192.168.100.1 origif: vmx0.99 @705 pass in quick on vmx0.5 route-to (vmx0.99 192.168.100.1) inet proto icmp from 10.0.5.0/24 to ! <a_Local4:13> keep state label "USER_RULE: ping to internet" label "id:1529164111" label "gw:WAN_GWV4" ridentifier 1529164111 tag qos_14039_default [ Evaluations: 5799 Packets: 1188 Bytes: 70392 States: 1 ] [ Inserted: uid 0 pid 16920 State Creations: 1 ] [ Last Active Time: Sun Feb 26 21:45:59 2023 ] @155 pass out route-to (vmx0.99 192.168.100.1) inet from 192.168.100.11 to ! 192.168.100.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself" ridentifier 1000015261 [ Evaluations: 313552 Packets: 18578109 Bytes: 942851185 States: 64 ] [ Inserted: uid 0 pid 16920 State Creations: 65 ] [ Last Active Time: Sun Feb 26 21:46:16 2023 ]
With the floating rule, both the download and upload speeds are correctly limited:
all icmp 1.0.0.1:1 <- 10.0.5.50:1 0:0 age 00:00:02, expires in 00:00:10, 3:3 pkts, 180:180 bytes, rule 706 id: 0b2ef26400000000 creatorid: 4da82510 gateway: 192.168.100.1 origif: vmx0.5 all icmp 192.168.100.11:55598 (10.0.5.50:1) -> 1.0.0.1:55598 0:0 age 00:00:02, expires in 00:00:10, 3:3 pkts, 180:180 bytes, rule 281 id: 0c2ef26400000000 creatorid: 4da82510 gateway: 192.168.100.1 origif: vmx0.99 @706 pass in quick on vmx0.5 route-to (vmx0.99 192.168.100.1) inet proto icmp from 10.0.5.0/24 to ! <a_Local4:13> keep state label "USER_RULE: ping to internet" label "id:1529164111" label "gw:WAN_GWV4" ridentifier 1529164111 tag qos_14039_default [ Evaluations: 5748 Packets: 1120 Bytes: 66312 States: 1 ] [ Inserted: uid 0 pid 99814 State Creations: 4 ] [ Last Active Time: Sun Feb 26 21:43:13 2023 ] @281 pass out quick on vmx0.99 route-to (vmx0.99 192.168.100.1) inet from any to ! <a_Local4:13> flags S/SA keep state label "USER_RULE: #14039 default" label "id:1677460989" label "gw:ISP1_STATIC" ridentifier 1677460989 dnqueue 3 tagged qos_14039_default [ Evaluations: 46067 Packets: 2124006 Bytes: 2207764474 States: 17 ] [ Inserted: uid 0 pid 99814 State Creations: 70 ] [ Last Active Time: Sun Feb 26 21:43:48 2023 ]
Without the floating rule and without route-to on the pass in rule, both the download and upload speeds are correctly limited:
all icmp 1.0.0.1:1 <- 10.0.5.50:1 0:0 age 00:00:01, expires in 00:00:10, 2:2 pkts, 120:120 bytes, rule 705 id: 0b5cf26400000000 creatorid: 4da82510 gateway: 0.0.0.0 origif: vmx0.5 all icmp 192.168.100.11:2227 (10.0.5.50:1) -> 1.0.0.1:2227 0:0 age 00:00:01, expires in 00:00:10, 2:2 pkts, 120:120 bytes, rule 155 id: 0c5cf26400000000 creatorid: 4da82510 gateway: 192.168.100.1 origif: vmx0.99 @705 pass in quick on vmx0.5 inet proto icmp from 10.0.5.0/24 to ! <a_Local4:13> keep state label "USER_RULE: ping to internet" label "id:1529164111" ridentifier 1529164111 tag qos_14039_default [ Evaluations: 51 Packets: 12 Bytes: 720 States: 0 ] [ Inserted: uid 0 pid 99857 State Creations: 2 ] [ Last Active Time: Sun Feb 26 22:05:09 2023 ] @155 pass out route-to (vmx0.99 192.168.100.1) inet from 192.168.100.11 to ! 192.168.100.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself" ridentifier 1000015261 [ Evaluations: 319499 Packets: 18989148 Bytes: 1384500604 States: 94 ] [ Inserted: uid 0 pid 99857 State Creations: 123 ] [ Last Active Time: Sun Feb 26 22:05:50 2023 ]
For reference, limiters were tested using speedtest.net with IPv4. In this setup, the download limit is applied via a match rule:
match in on { vmx0 vmx0.5 vmx0.50 vmx0.100 } inet from any to ! $a_Local4 tag "qos_default" ridentifier 1652043092 label "USER_RULE: QoS tag default IPv4" label "id:1652043092"
match in on { vmx0 vmx0.5 vmx0.10 vmx0.20 vmx0.50 vmx0.100 } inet from any to any tagged "qos_default" ridentifier 1652459212 dnqueue( 3,6) label "USER_RULE: QoS queue default IPv4" label "id:1652459212"
Updated by Jose Duarte about 1 year ago
I think in general you currently don't need more testers but can at least share that we are quite affected since we have Multi-WAN + GRE tunnels so the limiters + route-to are essencial for all our pfSense and pfSense+ firewalls.
The floating rule workaround is good to know though.
Updated by Marco Goetze about 1 year ago
- File limiter.png limiter.png added
Tried as Floating Rule but for me same outcome, the Limiter Diagnostic showing no limiter applied.
Updated by Kristof Provost 11 months ago
- Status changed from In Progress to Feedback
I've cherry picked this:
commit 89d6a1d11a369f5b5fcf9a2925a2064696970c0b (HEAD -> devel-main, origin/devel-main)
Author: Kristof Provost <kp@FreeBSD.org>
Date: Tue May 30 21:17:54 2023 +0200
pf: carry over rule actions from route-to rules
If we route-to (or dup-to/reply-to) we re-run pf_test(), which will also
create states for the connection.
This means that we may end up matching a different (i.e. not the state
that was created by the route-to rule) state, without the attributes
(such as dummynet pipes/queues) set by the route-to rule.
Address this by inheriting the pf_rule_actions from the route-to rule
while evaluating the connection again in pf_test(). That is, we set
default pf_rule_actions based on the route-to rule for the new
evaluation. The new rule may still overrule these, but if it does not
have such actions the route-to actions are applied.
Do the same for IPv6 rules in pf_test6()/pf_route6().
See also: https://redmine.pfsense.org/issues/14039
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D40340
(cherry picked from commit 9925aee0aaeccabd26f41625694a97b64185a59d)
to devel-main, which fixes this problem.
Updated by Jim Pingle 9 months ago
- Subject changed from Limiters do not limit upload traffic passed by policy-routing rules to Limiters have no effect on upload traffic passed by policy routing rules
Updating subject for release notes.
Updated by Mike McNabb 7 months ago
Marcos M wrote in #note-2:
The issue can be avoided by creating a floating rule that applies the upload limiter.
Without the floating rule, the download speed is limited and the upload speed is unlimited:
[...]With the floating rule, both the download and upload speeds are correctly limited:
[...]Without the floating rule and without
route-toon thepass inrule, both the download and upload speeds are correctly limited:
[...]For reference, limiters were tested using
speedtest.netwith IPv4. In this setup, the download limit is applied via a match rule:match in on { vmx0 vmx0.5 vmx0.50 vmx0.100 } inet from any to ! $a_Local4 tag "qos_default" ridentifier 1652043092 label "USER_RULE: QoS tag default IPv4" label "id:1652043092"
match in on { vmx0 vmx0.5 vmx0.10 vmx0.20 vmx0.50 vmx0.100 } inet from any to any tagged "qos_default" ridentifier 1652459212 dnqueue( 3,6) label "USER_RULE: QoS queue default IPv4" label "id:1652459212"
Sorry, I'm having a hard time following your workaround. I tried removing the limiters from the LAN rule and adding a match rule in floating but it still only applies to the Download.