Project

General

Profile

Actions

Regression #13420

closed

TCP traffic sourced from the firewall can only use the default gateway

Added by Steve Wheeler almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Category:
Rules / NAT
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
23.01
Release Notes:
Default
Affected Version:
2.6.0
Affected Architecture:
All

Description

Traffic sourced from the firewall itself will always open states on the interface with the default system route. Even though traffic may not actually leave via that interface.

This means that for TCP traffic it can be blocked by the default firewall rules as being out-of-state.

For example this 6100 has two WANs configured. Both are DHCP with gateways passed by the upstream server:

[22.05-RELEASE][root@6100-2.stevew.lan]/root: netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.21.16.1        UGS         ix3
127.0.0.1          link#10            UH          lo0
172.21.16.0/24     link#8             U           ix3
172.21.16.1        90:ec:77:0f:74:43  UHS         ix3
172.21.16.170      link#8             UHS         lo0
192.168.170.0/24   link#1             U          igc0
192.168.170.1      link#1             UHS         lo0
192.168.241.0/24   link#7             U           ix2
192.168.241.1      90:ec:77:0f:74:44  UHS         ix2
192.168.241.10     link#7             UHS         lo0

The default gateway is via WAN (ix3).

It can a device that is further upstream outside both the WAN subnets via either route:

[22.05-RELEASE][root@6100-2.stevew.lan]/root: ping -c 2 -S 172.21.16.170 192.168.2.32
PING 192.168.2.32 (192.168.2.32) from 172.21.16.170: 56 data bytes
64 bytes from 192.168.2.32: icmp_seq=0 ttl=62 time=1.064 ms
64 bytes from 192.168.2.32: icmp_seq=1 ttl=62 time=0.753 ms

--- 192.168.2.32 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.753/0.909/1.064/0.156 ms
[22.05-RELEASE][root@6100-2.stevew.lan]/root: ping -c 2 -S 192.168.241.10 192.168.2.32
PING 192.168.2.32 (192.168.2.32) from 192.168.241.10: 56 data bytes
64 bytes from 192.168.2.32: icmp_seq=0 ttl=61 time=1.274 ms
64 bytes from 192.168.2.32: icmp_seq=1 ttl=61 time=0.781 ms

--- 192.168.2.32 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.781/1.027/1.274/0.247 ms

However a TCP connection to the same target will only succeed via the default system route:

[22.05-RELEASE][root@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170
Connecting to host 192.168.2.32, port 5201
[  5] local 172.21.16.170 port 29501 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  92.0 MBytes   771 Mbits/sec    3    369 KBytes       
[  5]   1.00-2.00   sec  92.4 MBytes   775 Mbits/sec   17    389 KBytes       
[  5]   2.00-3.00   sec  93.3 MBytes   783 Mbits/sec    2    392 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   278 MBytes   776 Mbits/sec   22             sender
[  5]   0.00-3.37   sec   277 MBytes   689 Mbits/sec                  receiver

iperf Done.
[22.05-RELEASE][root@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10
Connecting to host 192.168.2.32, port 5201
[  5] local 192.168.241.10 port 32893 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec   137 KBytes  1.05 Mbits/sec    3   1.41 KBytes       
[  5]   1.06-2.01   sec  0.00 Bytes  0.00 bits/sec    2   1.41 KBytes       
[  5]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.01   sec   137 KBytes   372 Kbits/sec    5             sender
[  5]   0.00-3.22   sec   116 KBytes   295 Kbits/sec                  receiver

iperf Done.

On the firewall itself both those connections show as states on the WAN interface:

all tcp 172.21.16.170:17826 -> 192.168.2.32:5201       FIN_WAIT_2:FIN_WAIT_2
   [3379762330 + 2147352832] wscale 7  [1691518013 + 4278255872] wscale 7
   age 00:00:12, expires in 00:01:21, 18:15 pkts, 1380:1090 bytes, rule 85
   id: 931afc6200000003 creatorid: 6d1e6c6f gateway: 172.21.16.1
   origif: ix3
all tcp 192.168.241.10:10643 -> 192.168.2.32:5201       ESTABLISHED:CLOSING
   [879166104 + 838795520] wscale 7  [2257063987 + 3841851648] wscale 7
   age 00:00:06, expires in 00:14:57, 18:17 pkts, 1376:1174 bytes, rule 86
   id: 941afc6200000003 creatorid: 6d1e6c6f gateway: 192.168.241.1
   origif: ix3

all tcp 172.21.16.170:29501 -> 192.168.2.32:5201       TIME_WAIT:TIME_WAIT
   [4223141156 + 2280204800] wscale 7  [3053074132 + 65792] wscale 7
   age 00:00:12, expires in 00:01:21, 200932:91890 pkts, 301393701:4782292 bytes, rule 85
   id: ad1afc6200000001 creatorid: 6d1e6c6f gateway: 172.21.16.1
   origif: ix3
all tcp 192.168.241.10:32893 -> 192.168.2.32:5201       ESTABLISHED:SYN_SENT
   [1066512255 + 3856466432] wscale 7  [116550158 + 4294902016] wscale 7
   age 00:00:06, expires in 00:00:24, 92:22 pkts, 133701:1152 bytes, rule 86
   id: ae1afc6200000001 creatorid: 6d1e6c6f gateway: 192.168.241.1
   origif: ix3

The upstream router on WAN2 shows the traffic was correctly sent via that route:

all tcp 192.168.2.32:5201 <- 192.168.241.10:25352       FIN_WAIT_2:ESTABLISHED
   [3059127359 + 4278255872] wscale 7  [1873996070 + 2147090688] wscale 7
   age 00:10:57, expires in 00:05:00, 90:48 pkts, 130701:2624 bytes, rule 136
   id: 18c4f06200000000 creatorid: 8775c03c gateway: 0.0.0.0
   origif: ix2
all tcp 172.21.16.246:44264 (192.168.241.10:25352) -> 192.168.2.32:5201       ESTABLISHED:FIN_WAIT_2
   [1873996070 + 2147090688] wscale 7  [3059127359 + 4278255872] wscale 7
   age 00:10:57, expires in 00:05:00, 90:48 pkts, 130701:2624 bytes, rule 113
   id: 19c4f06200000000 creatorid: 8775c03c gateway: 172.21.16.1
   origif: ix3
all tcp 192.168.2.32:5201 <- 192.168.241.10:32893       FIN_WAIT_2:ESTABLISHED
   [133327374 + 4278255872] wscale 7  [895594879 + 2147090688] wscale 7
   age 00:03:33, expires in 00:12:25, 94:49 pkts, 136701:2664 bytes, rule 136
   id: 4444f16200000001 creatorid: 8775c03c gateway: 0.0.0.0
   origif: ix2
all tcp 172.21.16.246:5354 (192.168.241.10:32893) -> 192.168.2.32:5201       ESTABLISHED:FIN_WAIT_2
   [895594879 + 2147090688] wscale 7  [133327374 + 4278255872] wscale 7
   age 00:03:33, expires in 00:12:25, 94:49 pkts, 136701:2664 bytes, rule 113
   id: 4544f16200000001 creatorid: 8775c03c gateway: 172.21.16.1
   origif: ix3

The firewall log on the sending device shows that traffic is eventually blocked as the TCP flags do not match the open states:

Aug 17 00:20:47 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS
Aug 17 00:21:02 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS
Aug 17 00:21:11 6100-2 filterlog[39606]: 4,,,1000000103,ix2,match,block,in,4,0x0,,61,47134,0,DF,6,tcp,52,192.168.2.32,192.168.241.10,5201,32893,0,FA,241889798,2142331189,501,,nop;nop;TS
Aug 17 00:21:17 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS
Aug 17 00:21:34 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,RA,2142270373:2142271821,241889798,0,,nop;nop;TS

Tested in 22.05-rel


Files

6100-2_rules.txt (11.6 KB) 6100-2_rules.txt Steve Wheeler, 08/17/2022 06:45 AM
6100-2_ruleset.txt (10.3 KB) 6100-2_ruleset.txt Steve Wheeler, 08/17/2022 06:45 AM

Related issues

Related to Bug #12630: States are always created on the default gateway interface.Not a Bug

Actions
Related to Todo #15173: Add global option to set default PF State Policy (if-bound vs floating)ResolvedJim Pingle

Actions
Actions #1

Updated by Steve Wheeler almost 2 years ago

This only affects traffic sourced from the firewall itself. Policy routed traffic from other local subnets opens states as expected and passes.

Also see: https://redmine.pfsense.org/issues/12630

Actions #2

Updated by Steve Wheeler almost 2 years ago

Attached rules from the tested firewall in 22.05.

Actions #3

Updated by Steve Wheeler almost 2 years ago

Tested:
22.09.a.20220729.0600 - same behaviour
21.02.2-rel - same behaviour
21.02-rel - works as expected

[21.02-RELEASE][root@pfSense.home.arpa]/root: uname -a
FreeBSD 6100-2.stevew.lan 12.2-STABLE FreeBSD 12.2-STABLE 38a4c12973d(plus-devel-12) pfSense  amd64
[21.02-RELEASE][root@pfSense.home.arpa]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170
Connecting to host 192.168.2.32, port 5201
[  5] local 172.21.16.170 port 10765 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  93.4 MBytes   784 Mbits/sec  253   32.5 KBytes       
[  5]   1.00-2.00   sec  92.6 MBytes   776 Mbits/sec    2    421 KBytes       
[  5]   2.00-3.00   sec  93.3 MBytes   783 Mbits/sec   10    215 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   279 MBytes   781 Mbits/sec  265             sender
[  5]   0.00-3.22   sec   278 MBytes   724 Mbits/sec                  receiver

iperf Done.
[21.02-RELEASE][root@pfSense.home.arpa]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10
Connecting to host 192.168.2.32, port 5201
[  5] local 192.168.241.10 port 16148 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  83.5 MBytes   701 Mbits/sec   10    221 KBytes       
[  5]   1.00-2.00   sec  91.7 MBytes   769 Mbits/sec    4    207 KBytes       
[  5]   2.00-3.00   sec  91.0 MBytes   764 Mbits/sec   14    424 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   266 MBytes   744 Mbits/sec   28             sender
[  5]   0.00-3.22   sec   266 MBytes   692 Mbits/sec                  receiver

iperf Done.
Actions #4

Updated by Marcos M almost 2 years ago

  • Project changed from pfSense Plus to pfSense
  • Category changed from Rules / NAT to Rules / NAT
  • Target version deleted (23.01)
  • Affected Plus Version deleted (22.05)
  • Plus Target Version set to 22.11
Actions #5

Updated by Marcos M almost 2 years ago

  • Related to Bug #12630: States are always created on the default gateway interface. added
Actions #6

Updated by Steve Wheeler almost 2 years ago

  • Target version set to 2.7.0
  • Affected Version set to 2.6.0

Tested:
2.5.0 - Passes TCP traffic from both WANs
2.5.1 - Fails as described
2.5.2 - Fails as described
2.6.0 - Fails as described
2.7.0.a.20220812.0002 - Fails as described

Actions #7

Updated by Kristof Provost almost 2 years ago

I can reproduce the problem on a 22.09 snapshot, but not on a main-based image:

[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: env LD_LIBRARY_PATH=/root ./iperf3 -t 3 -c 10.0.2.1 -B 1.0.3.89
Connecting to host 10.0.2.1, port 5201
[  5] local 1.0.3.89 port 22009 connected to 10.0.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  76.8 MBytes   644 Mbits/sec    0    325 KBytes
[  5]   1.00-2.00   sec  76.8 MBytes   644 Mbits/sec    0    325 KBytes
[  5]   2.00-3.00   sec  76.8 MBytes   645 Mbits/sec    0    325 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   230 MBytes   644 Mbits/sec    0             sender
[  5]   0.00-3.00   sec   230 MBytes   643 Mbits/sec                  receiver

iperf Done.
[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: env LD_LIBRARY_PATH=/root ./iperf3 -t 3 -c 10.0.2.1 -B 1.0.2.78
Connecting to host 10.0.2.1, port 5201
[  5] local 1.0.2.78 port 6143 connected to 10.0.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  77.5 MBytes   650 Mbits/sec    0    325 KBytes
[  5]   1.00-2.00   sec  76.8 MBytes   644 Mbits/sec    0    325 KBytes
[  5]   2.00-3.00   sec  75.8 MBytes   636 Mbits/sec    0    325 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   230 MBytes   644 Mbits/sec    0             sender
[  5]   0.00-3.00   sec   230 MBytes   643 Mbits/sec                  receiver

iperf Done.
[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: uname -a
FreeBSD pfSense.home.arpa 14.0-CURRENT FreeBSD 14.0-CURRENT #0 plus-devel-main-n255891-f8ae608637e: Tue Aug 23 11:44:07 CEST 2022     root@nut:/usr/home/kp/netgate/crossbuild-main/obj/amd64/lg6m8yhq/usr/home/kp/netgate/crossbuild-main/sources/FreeBSD-src-kp-devel-main/amd64.amd64/sys/pfSense amd64

Given that, I do not intend to investigate further. We should merely re-test this prior to release to ensure it remains fixed.

Actions #8

Updated by Jim Pingle almost 2 years ago

  • Plus Target Version changed from 22.11 to 23.01
Actions #9

Updated by Jim Pingle over 1 year ago

  • Status changed from New to Feedback
  • Assignee set to Kristof Provost

Now that we are on main-based builds this needs retested/reconfirmed.

Actions #10

Updated by Steve Wheeler over 1 year ago

The same test works as expected in 23.01:

[23.01-BETA][admin@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170
Connecting to host 192.168.2.32, port 5201
[  5] local 172.21.16.170 port 7682 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  55.7 MBytes   468 Mbits/sec  716    378 KBytes       
[  5]   1.00-2.00   sec  91.6 MBytes   768 Mbits/sec   11    402 KBytes       
[  5]   2.00-3.00   sec  82.6 MBytes   693 Mbits/sec    2    371 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   230 MBytes   643 Mbits/sec  729             sender
[  5]   0.00-3.01   sec   228 MBytes   637 Mbits/sec                  receiver

iperf Done.
[23.01-BETA][admin@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10
Connecting to host 192.168.2.32, port 5201
[  5] local 192.168.241.10 port 9787 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  87.0 MBytes   730 Mbits/sec    6    376 KBytes       
[  5]   1.00-2.00   sec  93.8 MBytes   787 Mbits/sec    4    366 KBytes       
[  5]   2.00-3.00   sec  91.2 MBytes   765 Mbits/sec    5    321 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   272 MBytes   761 Mbits/sec   15             sender
[  5]   0.00-3.00   sec   271 MBytes   757 Mbits/sec                  receiver

iperf Done.

States are still opened on the default WAN interface though:

all tcp 192.168.241.10:22276 -> 192.168.2.32:5201       FIN_WAIT_2:FIN_WAIT_2
   [712900838 + 2147287296] wscale 7  [1242105029 + 65792] wscale 7
   age 00:00:03, expires in 00:01:30, 18:17 pkts, 1397:1192 bytes, rule 89
   id: bdd99c6300000000 creatorid: 6d1e6c6f gateway: 192.168.241.1
   origif: ix3
all tcp 192.168.241.10:35736 -> 192.168.2.32:5201       TIME_WAIT:TIME_WAIT
   [3804588798 + 2014259712] wscale 7  [3154521839 + 65792] wscale 7
   age 00:00:03, expires in 00:01:30, 186572:87987 pkts, 279845013:4581928 bytes, rule 89
   id: bed99c6300000000 creatorid: 6d1e6c6f gateway: 192.168.241.1
   origif: ix3
```

Tested:

23.01-BETA (amd64)
built on Fri Dec 16 06:05:35 UTC 2022
FreeBSD 14.0-CURRENT

Actions #11

Updated by Georgiy Tyutyunnik over 1 year ago

tested to the same result as Steve Wheeler - traffic flows correctly but states are present on the interface with default gateway.
tested on:
Version 2.7.0-DEVELOPMENT (amd64)
built on Wed Dec 21 19:50:43 UTC 2022
FreeBSD 14.0-CURRENT

Actions #12

Updated by Jim Pingle over 1 year ago

  • Status changed from Feedback to Resolved
Actions #13

Updated by Jim Pingle over 1 year ago

This is the intended behavior, so it's safe to close.

Actions #14

Updated by Steve Wheeler over 1 year ago

Works as expected in:

23.01-BETA (amd64)
built on Fri Dec 16 06:05:35 UTC 2022
FreeBSD 14.0-CURRENT

Actions #15

Updated by Marcos M 5 months ago

  • Related to Todo #15220: Handle ``route-to`` and ``reply-to`` states when using the ``if-bound`` state policy added
Actions #16

Updated by Marcos M 5 months ago

  • Related to Todo #15173: Add global option to set default PF State Policy (if-bound vs floating) added
Actions #17

Updated by Marcos M 5 months ago

  • Related to deleted (Todo #15220: Handle ``route-to`` and ``reply-to`` states when using the ``if-bound`` state policy)
Actions

Also available in: Atom PDF