Actions
Regression #13420
closed
TCP traffic sourced from the firewall can only use the default gateway
Start date:
Due date:
% Done:
0%
Estimated time:
Plus Target Version:
23.01
Release Notes:
Default
Affected Version:
2.6.0
Affected Architecture:
All
Description
Traffic sourced from the firewall itself will always open states on the interface with the default system route. Even though traffic may not actually leave via that interface.
This means that for TCP traffic it can be blocked by the default firewall rules as being out-of-state.
For example this 6100 has two WANs configured. Both are DHCP with gateways passed by the upstream server:
[22.05-RELEASE][root@6100-2.stevew.lan]/root: netstat -rn4 Routing tables Internet: Destination Gateway Flags Netif Expire default 172.21.16.1 UGS ix3 127.0.0.1 link#10 UH lo0 172.21.16.0/24 link#8 U ix3 172.21.16.1 90:ec:77:0f:74:43 UHS ix3 172.21.16.170 link#8 UHS lo0 192.168.170.0/24 link#1 U igc0 192.168.170.1 link#1 UHS lo0 192.168.241.0/24 link#7 U ix2 192.168.241.1 90:ec:77:0f:74:44 UHS ix2 192.168.241.10 link#7 UHS lo0
The default gateway is via WAN (ix3).
It can a device that is further upstream outside both the WAN subnets via either route:
[22.05-RELEASE][root@6100-2.stevew.lan]/root: ping -c 2 -S 172.21.16.170 192.168.2.32 PING 192.168.2.32 (192.168.2.32) from 172.21.16.170: 56 data bytes 64 bytes from 192.168.2.32: icmp_seq=0 ttl=62 time=1.064 ms 64 bytes from 192.168.2.32: icmp_seq=1 ttl=62 time=0.753 ms --- 192.168.2.32 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.753/0.909/1.064/0.156 ms [22.05-RELEASE][root@6100-2.stevew.lan]/root: ping -c 2 -S 192.168.241.10 192.168.2.32 PING 192.168.2.32 (192.168.2.32) from 192.168.241.10: 56 data bytes 64 bytes from 192.168.2.32: icmp_seq=0 ttl=61 time=1.274 ms 64 bytes from 192.168.2.32: icmp_seq=1 ttl=61 time=0.781 ms --- 192.168.2.32 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.781/1.027/1.274/0.247 ms
However a TCP connection to the same target will only succeed via the default system route:
[22.05-RELEASE][root@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170 Connecting to host 192.168.2.32, port 5201 [ 5] local 172.21.16.170 port 29501 connected to 192.168.2.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 92.0 MBytes 771 Mbits/sec 3 369 KBytes [ 5] 1.00-2.00 sec 92.4 MBytes 775 Mbits/sec 17 389 KBytes [ 5] 2.00-3.00 sec 93.3 MBytes 783 Mbits/sec 2 392 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 278 MBytes 776 Mbits/sec 22 sender [ 5] 0.00-3.37 sec 277 MBytes 689 Mbits/sec receiver iperf Done. [22.05-RELEASE][root@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10 Connecting to host 192.168.2.32, port 5201 [ 5] local 192.168.241.10 port 32893 connected to 192.168.2.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.06 sec 137 KBytes 1.05 Mbits/sec 3 1.41 KBytes [ 5] 1.06-2.01 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes [ 5] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.01 sec 137 KBytes 372 Kbits/sec 5 sender [ 5] 0.00-3.22 sec 116 KBytes 295 Kbits/sec receiver iperf Done.
On the firewall itself both those connections show as states on the WAN interface:
all tcp 172.21.16.170:17826 -> 192.168.2.32:5201 FIN_WAIT_2:FIN_WAIT_2 [3379762330 + 2147352832] wscale 7 [1691518013 + 4278255872] wscale 7 age 00:00:12, expires in 00:01:21, 18:15 pkts, 1380:1090 bytes, rule 85 id: 931afc6200000003 creatorid: 6d1e6c6f gateway: 172.21.16.1 origif: ix3 all tcp 192.168.241.10:10643 -> 192.168.2.32:5201 ESTABLISHED:CLOSING [879166104 + 838795520] wscale 7 [2257063987 + 3841851648] wscale 7 age 00:00:06, expires in 00:14:57, 18:17 pkts, 1376:1174 bytes, rule 86 id: 941afc6200000003 creatorid: 6d1e6c6f gateway: 192.168.241.1 origif: ix3 all tcp 172.21.16.170:29501 -> 192.168.2.32:5201 TIME_WAIT:TIME_WAIT [4223141156 + 2280204800] wscale 7 [3053074132 + 65792] wscale 7 age 00:00:12, expires in 00:01:21, 200932:91890 pkts, 301393701:4782292 bytes, rule 85 id: ad1afc6200000001 creatorid: 6d1e6c6f gateway: 172.21.16.1 origif: ix3 all tcp 192.168.241.10:32893 -> 192.168.2.32:5201 ESTABLISHED:SYN_SENT [1066512255 + 3856466432] wscale 7 [116550158 + 4294902016] wscale 7 age 00:00:06, expires in 00:00:24, 92:22 pkts, 133701:1152 bytes, rule 86 id: ae1afc6200000001 creatorid: 6d1e6c6f gateway: 192.168.241.1 origif: ix3
The upstream router on WAN2 shows the traffic was correctly sent via that route:
all tcp 192.168.2.32:5201 <- 192.168.241.10:25352 FIN_WAIT_2:ESTABLISHED [3059127359 + 4278255872] wscale 7 [1873996070 + 2147090688] wscale 7 age 00:10:57, expires in 00:05:00, 90:48 pkts, 130701:2624 bytes, rule 136 id: 18c4f06200000000 creatorid: 8775c03c gateway: 0.0.0.0 origif: ix2 all tcp 172.21.16.246:44264 (192.168.241.10:25352) -> 192.168.2.32:5201 ESTABLISHED:FIN_WAIT_2 [1873996070 + 2147090688] wscale 7 [3059127359 + 4278255872] wscale 7 age 00:10:57, expires in 00:05:00, 90:48 pkts, 130701:2624 bytes, rule 113 id: 19c4f06200000000 creatorid: 8775c03c gateway: 172.21.16.1 origif: ix3 all tcp 192.168.2.32:5201 <- 192.168.241.10:32893 FIN_WAIT_2:ESTABLISHED [133327374 + 4278255872] wscale 7 [895594879 + 2147090688] wscale 7 age 00:03:33, expires in 00:12:25, 94:49 pkts, 136701:2664 bytes, rule 136 id: 4444f16200000001 creatorid: 8775c03c gateway: 0.0.0.0 origif: ix2 all tcp 172.21.16.246:5354 (192.168.241.10:32893) -> 192.168.2.32:5201 ESTABLISHED:FIN_WAIT_2 [895594879 + 2147090688] wscale 7 [133327374 + 4278255872] wscale 7 age 00:03:33, expires in 00:12:25, 94:49 pkts, 136701:2664 bytes, rule 113 id: 4544f16200000001 creatorid: 8775c03c gateway: 172.21.16.1 origif: ix3
The firewall log on the sending device shows that traffic is eventually blocked as the TCP flags do not match the open states:
Aug 17 00:20:47 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS Aug 17 00:21:02 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS Aug 17 00:21:11 6100-2 filterlog[39606]: 4,,,1000000103,ix2,match,block,in,4,0x0,,61,47134,0,DF,6,tcp,52,192.168.2.32,192.168.241.10,5201,32893,0,FA,241889798,2142331189,501,,nop;nop;TS Aug 17 00:21:17 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS Aug 17 00:21:34 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,RA,2142270373:2142271821,241889798,0,,nop;nop;TS
Tested in 22.05-rel
Files
Related issues
Updated by Steve Wheeler over 2 years ago
This only affects traffic sourced from the firewall itself. Policy routed traffic from other local subnets opens states as expected and passes.
Also see: https://redmine.pfsense.org/issues/12630
Updated by Steve Wheeler over 2 years ago
- File 6100-2_rules.txt 6100-2_rules.txt added
- File 6100-2_ruleset.txt 6100-2_ruleset.txt added
Attached rules from the tested firewall in 22.05.
Updated by Steve Wheeler over 2 years ago
Tested:
22.09.a.20220729.0600 - same behaviour
21.02.2-rel - same behaviour
21.02-rel - works as expected
[21.02-RELEASE][root@pfSense.home.arpa]/root: uname -a FreeBSD 6100-2.stevew.lan 12.2-STABLE FreeBSD 12.2-STABLE 38a4c12973d(plus-devel-12) pfSense amd64 [21.02-RELEASE][root@pfSense.home.arpa]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170 Connecting to host 192.168.2.32, port 5201 [ 5] local 172.21.16.170 port 10765 connected to 192.168.2.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 93.4 MBytes 784 Mbits/sec 253 32.5 KBytes [ 5] 1.00-2.00 sec 92.6 MBytes 776 Mbits/sec 2 421 KBytes [ 5] 2.00-3.00 sec 93.3 MBytes 783 Mbits/sec 10 215 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 279 MBytes 781 Mbits/sec 265 sender [ 5] 0.00-3.22 sec 278 MBytes 724 Mbits/sec receiver iperf Done. [21.02-RELEASE][root@pfSense.home.arpa]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10 Connecting to host 192.168.2.32, port 5201 [ 5] local 192.168.241.10 port 16148 connected to 192.168.2.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 83.5 MBytes 701 Mbits/sec 10 221 KBytes [ 5] 1.00-2.00 sec 91.7 MBytes 769 Mbits/sec 4 207 KBytes [ 5] 2.00-3.00 sec 91.0 MBytes 764 Mbits/sec 14 424 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 266 MBytes 744 Mbits/sec 28 sender [ 5] 0.00-3.22 sec 266 MBytes 692 Mbits/sec receiver iperf Done.
Updated by Marcos M over 2 years ago
- Project changed from pfSense Plus to pfSense
- Category changed from Rules / NAT to Rules / NAT
- Target version deleted (
23.01) - Affected Plus Version deleted (
22.05) - Plus Target Version set to 22.11
Updated by Marcos M over 2 years ago
- Related to Bug #12630: States are always created on the default gateway interface. added
Updated by Steve Wheeler over 2 years ago
- Target version set to 2.7.0
- Affected Version set to 2.6.0
Tested:
2.5.0 - Passes TCP traffic from both WANs
2.5.1 - Fails as described
2.5.2 - Fails as described
2.6.0 - Fails as described
2.7.0.a.20220812.0002 - Fails as described
Updated by Kristof Provost over 2 years ago
I can reproduce the problem on a 22.09 snapshot, but not on a main-based image:
[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: env LD_LIBRARY_PATH=/root ./iperf3 -t 3 -c 10.0.2.1 -B 1.0.3.89 Connecting to host 10.0.2.1, port 5201 [ 5] local 1.0.3.89 port 22009 connected to 10.0.2.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 76.8 MBytes 644 Mbits/sec 0 325 KBytes [ 5] 1.00-2.00 sec 76.8 MBytes 644 Mbits/sec 0 325 KBytes [ 5] 2.00-3.00 sec 76.8 MBytes 645 Mbits/sec 0 325 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 230 MBytes 644 Mbits/sec 0 sender [ 5] 0.00-3.00 sec 230 MBytes 643 Mbits/sec receiver iperf Done. [22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: env LD_LIBRARY_PATH=/root ./iperf3 -t 3 -c 10.0.2.1 -B 1.0.2.78 Connecting to host 10.0.2.1, port 5201 [ 5] local 1.0.2.78 port 6143 connected to 10.0.2.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 77.5 MBytes 650 Mbits/sec 0 325 KBytes [ 5] 1.00-2.00 sec 76.8 MBytes 644 Mbits/sec 0 325 KBytes [ 5] 2.00-3.00 sec 75.8 MBytes 636 Mbits/sec 0 325 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 230 MBytes 644 Mbits/sec 0 sender [ 5] 0.00-3.00 sec 230 MBytes 643 Mbits/sec receiver iperf Done. [22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: uname -a FreeBSD pfSense.home.arpa 14.0-CURRENT FreeBSD 14.0-CURRENT #0 plus-devel-main-n255891-f8ae608637e: Tue Aug 23 11:44:07 CEST 2022 root@nut:/usr/home/kp/netgate/crossbuild-main/obj/amd64/lg6m8yhq/usr/home/kp/netgate/crossbuild-main/sources/FreeBSD-src-kp-devel-main/amd64.amd64/sys/pfSense amd64
Given that, I do not intend to investigate further. We should merely re-test this prior to release to ensure it remains fixed.
Updated by Jim Pingle about 2 years ago
- Plus Target Version changed from 22.11 to 23.01
Updated by Jim Pingle about 2 years ago
- Status changed from New to Feedback
- Assignee set to Kristof Provost
Now that we are on main-based builds this needs retested/reconfirmed.
Updated by Steve Wheeler almost 2 years ago
The same test works as expected in 23.01:
[23.01-BETA][admin@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170 Connecting to host 192.168.2.32, port 5201 [ 5] local 172.21.16.170 port 7682 connected to 192.168.2.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 55.7 MBytes 468 Mbits/sec 716 378 KBytes [ 5] 1.00-2.00 sec 91.6 MBytes 768 Mbits/sec 11 402 KBytes [ 5] 2.00-3.00 sec 82.6 MBytes 693 Mbits/sec 2 371 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 230 MBytes 643 Mbits/sec 729 sender [ 5] 0.00-3.01 sec 228 MBytes 637 Mbits/sec receiver iperf Done. [23.01-BETA][admin@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10 Connecting to host 192.168.2.32, port 5201 [ 5] local 192.168.241.10 port 9787 connected to 192.168.2.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 87.0 MBytes 730 Mbits/sec 6 376 KBytes [ 5] 1.00-2.00 sec 93.8 MBytes 787 Mbits/sec 4 366 KBytes [ 5] 2.00-3.00 sec 91.2 MBytes 765 Mbits/sec 5 321 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.00 sec 272 MBytes 761 Mbits/sec 15 sender [ 5] 0.00-3.00 sec 271 MBytes 757 Mbits/sec receiver iperf Done.
States are still opened on the default WAN interface though:
all tcp 192.168.241.10:22276 -> 192.168.2.32:5201 FIN_WAIT_2:FIN_WAIT_2 [712900838 + 2147287296] wscale 7 [1242105029 + 65792] wscale 7 age 00:00:03, expires in 00:01:30, 18:17 pkts, 1397:1192 bytes, rule 89 id: bdd99c6300000000 creatorid: 6d1e6c6f gateway: 192.168.241.1 origif: ix3 all tcp 192.168.241.10:35736 -> 192.168.2.32:5201 TIME_WAIT:TIME_WAIT [3804588798 + 2014259712] wscale 7 [3154521839 + 65792] wscale 7 age 00:00:03, expires in 00:01:30, 186572:87987 pkts, 279845013:4581928 bytes, rule 89 id: bed99c6300000000 creatorid: 6d1e6c6f gateway: 192.168.241.1 origif: ix3 ```
Tested:
23.01-BETA (amd64) built on Fri Dec 16 06:05:35 UTC 2022 FreeBSD 14.0-CURRENT
Updated by Georgiy Tyutyunnik almost 2 years ago
tested to the same result as Steve Wheeler - traffic flows correctly but states are present on the interface with default gateway.
tested on:
Version 2.7.0-DEVELOPMENT (amd64)
built on Wed Dec 21 19:50:43 UTC 2022
FreeBSD 14.0-CURRENT
Updated by Jim Pingle almost 2 years ago
- Status changed from Feedback to Resolved
Updated by Jim Pingle almost 2 years ago
This is the intended behavior, so it's safe to close.
Updated by Steve Wheeler almost 2 years ago
Works as expected in:
23.01-BETA (amd64) built on Fri Dec 16 06:05:35 UTC 2022 FreeBSD 14.0-CURRENT
Updated by Marcos M 10 months ago
- Related to Todo #15220: Handle ``route-to`` and ``reply-to`` states when using the ``if-bound`` state policy added
Updated by Marcos M 10 months ago
- Related to Todo #15173: Add global option to set default PF State Policy (if-bound vs floating) added
Updated by Marcos M 10 months ago
- Related to deleted (Todo #15220: Handle ``route-to`` and ``reply-to`` states when using the ``if-bound`` state policy)
Actions