Project

General

Profile

Actions

Regression #14039

closed

Limiters have no effect on upload traffic passed by policy routing rules

Added by Marcos M about 1 year ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Category:
Traffic Shaper (Limiters)
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
23.09
Release Notes:
Default
Affected Version:
2.7.0
Affected Architecture:
All

Description

Upload traffic is not limited if the rule passing the traffic uses route-to. This last worked in pfSense+ 22.01 and pfSense CE 2.6.0. See https://redmine.pfsense.org/issues/13026#note-15:

Essentially what happens is that we have two states:

all tcp 10.0.2.1:5201 <- 192.168.1.100:44607       ESTABLISHED:ESTABLISHED
   [2078351244 + 3221291264] wscale 6  [1276678361 + 2419064832] wscale 6
   age 00:00:03, expires in 24:00:00, 122313:60993 pkts, 183465201:3171644 bytes, rule 81
   id: a5627f6300000000 creatorid: a17f7a2e gateway: 1.0.2.1
   origif: vtnet2
all tcp 1.0.2.78:50878 (192.168.1.100:44607) -> 10.0.2.1:5201       ESTABLISHED:ESTABLISHED
   [1276678361 + 2419064832] wscale 6  [2078351244 + 3221291264] wscale 6
   age 00:00:03, expires in 24:00:00, 122313:60993 pkts, 183465201:3171644 bytes, rule 77
   id: a6627f6300000000 creatorid: a17f7a2e gateway: 1.0.2.1
   origif: vtnet0

The first state is created by the rule with the limiter, but because that rule also does route-to the packet is passed through pf_test() a second time, which creates the second state. That second state is created by a rule which doesn't have the limiter associated, and that means that when it matches the limiter is not applied. It's that second state that ends up matching incoming packets, so the limiter doesn't get applied there.


Files

limiter.png (108 KB) limiter.png Marco Goetze, 03/25/2023 04:53 AM
Actions #1

Updated by Marcos M about 1 year ago

  • Description updated (diff)
Actions #2

Updated by Marcos M about 1 year ago

The issue can be avoided by creating a floating rule that applies the upload limiter.

Without the floating rule, the download speed is limited and the upload speed is unlimited:

all icmp 1.0.0.1:1 <- 10.0.5.50:1       0:0
   age 00:00:07, expires in 00:00:09, 7:7 pkts, 420:420 bytes, rule 705
   id: 2734f26400000000 creatorid: 4da82510 gateway: 192.168.100.1
   origif: vmx0.5
all icmp 192.168.100.11:36640 (10.0.5.50:1) -> 1.0.0.1:36640       0:0
   age 00:00:07, expires in 00:00:09, 7:7 pkts, 420:420 bytes, rule 155
   id: 2834f26400000000 creatorid: 4da82510 gateway: 192.168.100.1
   origif: vmx0.99

@705 pass in quick on vmx0.5 route-to (vmx0.99 192.168.100.1) inet proto icmp from 10.0.5.0/24 to ! <a_Local4:13> keep state label "USER_RULE: ping to internet" label "id:1529164111" label "gw:WAN_GWV4" ridentifier 1529164111 tag qos_14039_default
  [ Evaluations: 5799      Packets: 1188      Bytes: 70392       States: 1     ]
  [ Inserted: uid 0 pid 16920 State Creations: 1     ]
  [ Last Active Time: Sun Feb 26 21:45:59 2023 ]
@155 pass out route-to (vmx0.99 192.168.100.1) inet from 192.168.100.11 to ! 192.168.100.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself" ridentifier 1000015261
  [ Evaluations: 313552    Packets: 18578109  Bytes: 942851185   States: 64    ]
  [ Inserted: uid 0 pid 16920 State Creations: 65    ]
  [ Last Active Time: Sun Feb 26 21:46:16 2023 ]

With the floating rule, both the download and upload speeds are correctly limited:

all icmp 1.0.0.1:1 <- 10.0.5.50:1       0:0
   age 00:00:02, expires in 00:00:10, 3:3 pkts, 180:180 bytes, rule 706
   id: 0b2ef26400000000 creatorid: 4da82510 gateway: 192.168.100.1
   origif: vmx0.5
all icmp 192.168.100.11:55598 (10.0.5.50:1) -> 1.0.0.1:55598       0:0
   age 00:00:02, expires in 00:00:10, 3:3 pkts, 180:180 bytes, rule 281
   id: 0c2ef26400000000 creatorid: 4da82510 gateway: 192.168.100.1
   origif: vmx0.99

@706 pass in quick on vmx0.5 route-to (vmx0.99 192.168.100.1) inet proto icmp from 10.0.5.0/24 to ! <a_Local4:13> keep state label "USER_RULE: ping to internet" label "id:1529164111" label "gw:WAN_GWV4" ridentifier 1529164111 tag qos_14039_default
  [ Evaluations: 5748      Packets: 1120      Bytes: 66312       States: 1     ]
  [ Inserted: uid 0 pid 99814 State Creations: 4     ]
  [ Last Active Time: Sun Feb 26 21:43:13 2023 ]
@281 pass out quick on vmx0.99 route-to (vmx0.99 192.168.100.1) inet from any to ! <a_Local4:13> flags S/SA keep state label "USER_RULE: #14039 default" label "id:1677460989" label "gw:ISP1_STATIC" ridentifier 1677460989 dnqueue 3 tagged qos_14039_default
  [ Evaluations: 46067     Packets: 2124006   Bytes: 2207764474  States: 17    ]
  [ Inserted: uid 0 pid 99814 State Creations: 70    ]
  [ Last Active Time: Sun Feb 26 21:43:48 2023 ]

Without the floating rule and without route-to on the pass in rule, both the download and upload speeds are correctly limited:

all icmp 1.0.0.1:1 <- 10.0.5.50:1       0:0
   age 00:00:01, expires in 00:00:10, 2:2 pkts, 120:120 bytes, rule 705
   id: 0b5cf26400000000 creatorid: 4da82510 gateway: 0.0.0.0
   origif: vmx0.5
all icmp 192.168.100.11:2227 (10.0.5.50:1) -> 1.0.0.1:2227       0:0
   age 00:00:01, expires in 00:00:10, 2:2 pkts, 120:120 bytes, rule 155
   id: 0c5cf26400000000 creatorid: 4da82510 gateway: 192.168.100.1
   origif: vmx0.99

@705 pass in quick on vmx0.5 inet proto icmp from 10.0.5.0/24 to ! <a_Local4:13> keep state label "USER_RULE: ping to internet" label "id:1529164111" ridentifier 1529164111 tag qos_14039_default
  [ Evaluations: 51        Packets: 12        Bytes: 720         States: 0     ]
  [ Inserted: uid 0 pid 99857 State Creations: 2     ]
  [ Last Active Time: Sun Feb 26 22:05:09 2023 ]
@155 pass out route-to (vmx0.99 192.168.100.1) inet from 192.168.100.11 to ! 192.168.100.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself" ridentifier 1000015261
  [ Evaluations: 319499    Packets: 18989148  Bytes: 1384500604  States: 94    ]
  [ Inserted: uid 0 pid 99857 State Creations: 123   ]
  [ Last Active Time: Sun Feb 26 22:05:50 2023 ]

For reference, limiters were tested using speedtest.net with IPv4. In this setup, the download limit is applied via a match rule:

match in on { vmx0 vmx0.5 vmx0.50 vmx0.100 } inet from any to ! $a_Local4 tag "qos_default" ridentifier 1652043092 label "USER_RULE: QoS tag default IPv4" label "id:1652043092"
match in on { vmx0 vmx0.5 vmx0.10 vmx0.20 vmx0.50 vmx0.100 } inet from any to any tagged "qos_default" ridentifier 1652459212 dnqueue( 3,6) label "USER_RULE: QoS queue default IPv4" label "id:1652459212"

Actions #3

Updated by Jose Duarte about 1 year ago

I think in general you currently don't need more testers but can at least share that we are quite affected since we have Multi-WAN + GRE tunnels so the limiters + route-to are essencial for all our pfSense and pfSense+ firewalls.
The floating rule workaround is good to know though.

Actions #4

Updated by Marco Goetze about 1 year ago

Tried as Floating Rule but for me same outcome, the Limiter Diagnostic showing no limiter applied.

Actions #5

Updated by Marcos M 11 months ago

  • Status changed from Confirmed to In Progress
  • Assignee set to Kristof Provost
  • Target version set to 2.7.0
  • Plus Target Version set to 23.09
  • Affected Architecture All added
Actions #7

Updated by Marcos M 11 months ago

  • Private changed from Yes to No
Actions #8

Updated by Kristof Provost 11 months ago

  • Status changed from In Progress to Feedback

I've cherry picked this:

commit 89d6a1d11a369f5b5fcf9a2925a2064696970c0b (HEAD -> devel-main, origin/devel-main)
Author: Kristof Provost <kp@FreeBSD.org>
Date:   Tue May 30 21:17:54 2023 +0200

    pf: carry over rule actions from route-to rules

    If we route-to (or dup-to/reply-to) we re-run pf_test(), which will also
    create states for the connection.
    This means that we may end up matching a different (i.e. not the state
    that was created by the route-to rule) state, without the attributes
    (such as dummynet pipes/queues) set by the route-to rule.

    Address this by inheriting the pf_rule_actions from the route-to rule
    while evaluating the connection again in pf_test(). That is, we set
    default pf_rule_actions based on the route-to rule for the new
    evaluation. The new rule may still overrule these, but if it does not
    have such actions the route-to actions are applied.

    Do the same for IPv6 rules in pf_test6()/pf_route6().

    See also:       https://redmine.pfsense.org/issues/14039
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D40340

    (cherry picked from commit 9925aee0aaeccabd26f41625694a97b64185a59d)

to devel-main, which fixes this problem.

Actions #9

Updated by Marcos M 11 months ago

  • Status changed from Feedback to Resolved

Tested fix - now works.

Actions #10

Updated by Jim Pingle 8 months ago

  • Subject changed from Limiters do not limit upload traffic passed by policy-routing rules to Limiters have no effect on upload traffic passed by policy routing rules

Updating subject for release notes.

Actions #11

Updated by Mike McNabb 7 months ago

Marcos M wrote in #note-2:

The issue can be avoided by creating a floating rule that applies the upload limiter.

Without the floating rule, the download speed is limited and the upload speed is unlimited:
[...]

With the floating rule, both the download and upload speeds are correctly limited:
[...]

Without the floating rule and without route-to on the pass in rule, both the download and upload speeds are correctly limited:
[...]

For reference, limiters were tested using speedtest.net with IPv4. In this setup, the download limit is applied via a match rule:

match in on { vmx0 vmx0.5 vmx0.50 vmx0.100 } inet from any to ! $a_Local4 tag "qos_default" ridentifier 1652043092 label "USER_RULE: QoS tag default IPv4" label "id:1652043092"
match in on { vmx0 vmx0.5 vmx0.10 vmx0.20 vmx0.50 vmx0.100 } inet from any to any tagged "qos_default" ridentifier 1652459212 dnqueue( 3,6) label "USER_RULE: QoS queue default IPv4" label "id:1652459212"

Sorry, I'm having a hard time following your workaround. I tried removing the limiters from the LAN rule and adding a match rule in floating but it still only applies to the Download.

Actions

Also available in: Atom PDF