Project

General

Profile

Actions

Regression #13420

open

TCP traffic sourced from the firewall can only use the default gateway

Added by Steve Wheeler about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Rules / NAT
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
22.11
Release Notes:
Default
Affected Version:
2.6.0
Affected Architecture:
All

Description

Traffic sourced from the firewall itself will always open states on the interface with the default system route. Even though traffic may not actually leave via that interface.

This means that for TCP traffic it can be blocked by the default firewall rules as being out-of-state.

For example this 6100 has two WANs configured. Both are DHCP with gateways passed by the upstream server:

[22.05-RELEASE][root@6100-2.stevew.lan]/root: netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.21.16.1        UGS         ix3
127.0.0.1          link#10            UH          lo0
172.21.16.0/24     link#8             U           ix3
172.21.16.1        90:ec:77:0f:74:43  UHS         ix3
172.21.16.170      link#8             UHS         lo0
192.168.170.0/24   link#1             U          igc0
192.168.170.1      link#1             UHS         lo0
192.168.241.0/24   link#7             U           ix2
192.168.241.1      90:ec:77:0f:74:44  UHS         ix2
192.168.241.10     link#7             UHS         lo0

The default gateway is via WAN (ix3).

It can a device that is further upstream outside both the WAN subnets via either route:

[22.05-RELEASE][root@6100-2.stevew.lan]/root: ping -c 2 -S 172.21.16.170 192.168.2.32
PING 192.168.2.32 (192.168.2.32) from 172.21.16.170: 56 data bytes
64 bytes from 192.168.2.32: icmp_seq=0 ttl=62 time=1.064 ms
64 bytes from 192.168.2.32: icmp_seq=1 ttl=62 time=0.753 ms

--- 192.168.2.32 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.753/0.909/1.064/0.156 ms
[22.05-RELEASE][root@6100-2.stevew.lan]/root: ping -c 2 -S 192.168.241.10 192.168.2.32
PING 192.168.2.32 (192.168.2.32) from 192.168.241.10: 56 data bytes
64 bytes from 192.168.2.32: icmp_seq=0 ttl=61 time=1.274 ms
64 bytes from 192.168.2.32: icmp_seq=1 ttl=61 time=0.781 ms

--- 192.168.2.32 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.781/1.027/1.274/0.247 ms

However a TCP connection to the same target will only succeed via the default system route:

[22.05-RELEASE][root@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170
Connecting to host 192.168.2.32, port 5201
[  5] local 172.21.16.170 port 29501 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  92.0 MBytes   771 Mbits/sec    3    369 KBytes       
[  5]   1.00-2.00   sec  92.4 MBytes   775 Mbits/sec   17    389 KBytes       
[  5]   2.00-3.00   sec  93.3 MBytes   783 Mbits/sec    2    392 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   278 MBytes   776 Mbits/sec   22             sender
[  5]   0.00-3.37   sec   277 MBytes   689 Mbits/sec                  receiver

iperf Done.
[22.05-RELEASE][root@6100-2.stevew.lan]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10
Connecting to host 192.168.2.32, port 5201
[  5] local 192.168.241.10 port 32893 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec   137 KBytes  1.05 Mbits/sec    3   1.41 KBytes       
[  5]   1.06-2.01   sec  0.00 Bytes  0.00 bits/sec    2   1.41 KBytes       
[  5]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.01   sec   137 KBytes   372 Kbits/sec    5             sender
[  5]   0.00-3.22   sec   116 KBytes   295 Kbits/sec                  receiver

iperf Done.

On the firewall itself both those connections show as states on the WAN interface:

all tcp 172.21.16.170:17826 -> 192.168.2.32:5201       FIN_WAIT_2:FIN_WAIT_2
   [3379762330 + 2147352832] wscale 7  [1691518013 + 4278255872] wscale 7
   age 00:00:12, expires in 00:01:21, 18:15 pkts, 1380:1090 bytes, rule 85
   id: 931afc6200000003 creatorid: 6d1e6c6f gateway: 172.21.16.1
   origif: ix3
all tcp 192.168.241.10:10643 -> 192.168.2.32:5201       ESTABLISHED:CLOSING
   [879166104 + 838795520] wscale 7  [2257063987 + 3841851648] wscale 7
   age 00:00:06, expires in 00:14:57, 18:17 pkts, 1376:1174 bytes, rule 86
   id: 941afc6200000003 creatorid: 6d1e6c6f gateway: 192.168.241.1
   origif: ix3

all tcp 172.21.16.170:29501 -> 192.168.2.32:5201       TIME_WAIT:TIME_WAIT
   [4223141156 + 2280204800] wscale 7  [3053074132 + 65792] wscale 7
   age 00:00:12, expires in 00:01:21, 200932:91890 pkts, 301393701:4782292 bytes, rule 85
   id: ad1afc6200000001 creatorid: 6d1e6c6f gateway: 172.21.16.1
   origif: ix3
all tcp 192.168.241.10:32893 -> 192.168.2.32:5201       ESTABLISHED:SYN_SENT
   [1066512255 + 3856466432] wscale 7  [116550158 + 4294902016] wscale 7
   age 00:00:06, expires in 00:00:24, 92:22 pkts, 133701:1152 bytes, rule 86
   id: ae1afc6200000001 creatorid: 6d1e6c6f gateway: 192.168.241.1
   origif: ix3

The upstream router on WAN2 shows the traffic was correctly sent via that route:

all tcp 192.168.2.32:5201 <- 192.168.241.10:25352       FIN_WAIT_2:ESTABLISHED
   [3059127359 + 4278255872] wscale 7  [1873996070 + 2147090688] wscale 7
   age 00:10:57, expires in 00:05:00, 90:48 pkts, 130701:2624 bytes, rule 136
   id: 18c4f06200000000 creatorid: 8775c03c gateway: 0.0.0.0
   origif: ix2
all tcp 172.21.16.246:44264 (192.168.241.10:25352) -> 192.168.2.32:5201       ESTABLISHED:FIN_WAIT_2
   [1873996070 + 2147090688] wscale 7  [3059127359 + 4278255872] wscale 7
   age 00:10:57, expires in 00:05:00, 90:48 pkts, 130701:2624 bytes, rule 113
   id: 19c4f06200000000 creatorid: 8775c03c gateway: 172.21.16.1
   origif: ix3
all tcp 192.168.2.32:5201 <- 192.168.241.10:32893       FIN_WAIT_2:ESTABLISHED
   [133327374 + 4278255872] wscale 7  [895594879 + 2147090688] wscale 7
   age 00:03:33, expires in 00:12:25, 94:49 pkts, 136701:2664 bytes, rule 136
   id: 4444f16200000001 creatorid: 8775c03c gateway: 0.0.0.0
   origif: ix2
all tcp 172.21.16.246:5354 (192.168.241.10:32893) -> 192.168.2.32:5201       ESTABLISHED:FIN_WAIT_2
   [895594879 + 2147090688] wscale 7  [133327374 + 4278255872] wscale 7
   age 00:03:33, expires in 00:12:25, 94:49 pkts, 136701:2664 bytes, rule 113
   id: 4544f16200000001 creatorid: 8775c03c gateway: 172.21.16.1
   origif: ix3

The firewall log on the sending device shows that traffic is eventually blocked as the TCP flags do not match the open states:

Aug 17 00:20:47 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS
Aug 17 00:21:02 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS
Aug 17 00:21:11 6100-2 filterlog[39606]: 4,,,1000000103,ix2,match,block,in,4,0x0,,61,47134,0,DF,6,tcp,52,192.168.2.32,192.168.241.10,5201,32893,0,FA,241889798,2142331189,501,,nop;nop;TS
Aug 17 00:21:17 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,A,2142270373:2142271821,241889798,514,,nop;nop;TS
Aug 17 00:21:34 6100-2 filterlog[39606]: 5,,,1000000104,ix3,match,block,out,4,0x0,,64,0,0,DF,6,tcp,1500,192.168.241.10,192.168.2.32,32893,5201,1448,RA,2142270373:2142271821,241889798,0,,nop;nop;TS

Tested in 22.05-rel


Files

6100-2_rules.txt (11.6 KB) 6100-2_rules.txt Steve Wheeler, 08/17/2022 06:45 AM
6100-2_ruleset.txt (10.3 KB) 6100-2_ruleset.txt Steve Wheeler, 08/17/2022 06:45 AM

Related issues

Related to Bug #12630: States are always created on the default gateway interface.Not a Bug

Actions
Actions #1

Updated by Steve Wheeler about 1 month ago

This only affects traffic sourced from the firewall itself. Policy routed traffic from other local subnets opens states as expected and passes.

Also see: https://redmine.pfsense.org/issues/12630

Actions #2

Updated by Steve Wheeler about 1 month ago

Attached rules from the tested firewall in 22.05.

Actions #3

Updated by Steve Wheeler about 1 month ago

Tested:
22.09.a.20220729.0600 - same behaviour
21.02.2-rel - same behaviour
21.02-rel - works as expected

[21.02-RELEASE][root@pfSense.home.arpa]/root: uname -a
FreeBSD 6100-2.stevew.lan 12.2-STABLE FreeBSD 12.2-STABLE 38a4c12973d(plus-devel-12) pfSense  amd64
[21.02-RELEASE][root@pfSense.home.arpa]/root: iperf3 -c 192.168.2.32 -t 3 -B 172.21.16.170
Connecting to host 192.168.2.32, port 5201
[  5] local 172.21.16.170 port 10765 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  93.4 MBytes   784 Mbits/sec  253   32.5 KBytes       
[  5]   1.00-2.00   sec  92.6 MBytes   776 Mbits/sec    2    421 KBytes       
[  5]   2.00-3.00   sec  93.3 MBytes   783 Mbits/sec   10    215 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   279 MBytes   781 Mbits/sec  265             sender
[  5]   0.00-3.22   sec   278 MBytes   724 Mbits/sec                  receiver

iperf Done.
[21.02-RELEASE][root@pfSense.home.arpa]/root: iperf3 -c 192.168.2.32 -t 3 -B 192.168.241.10
Connecting to host 192.168.2.32, port 5201
[  5] local 192.168.241.10 port 16148 connected to 192.168.2.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  83.5 MBytes   701 Mbits/sec   10    221 KBytes       
[  5]   1.00-2.00   sec  91.7 MBytes   769 Mbits/sec    4    207 KBytes       
[  5]   2.00-3.00   sec  91.0 MBytes   764 Mbits/sec   14    424 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   266 MBytes   744 Mbits/sec   28             sender
[  5]   0.00-3.22   sec   266 MBytes   692 Mbits/sec                  receiver

iperf Done.
Actions #4

Updated by Marcos M about 1 month ago

  • Project changed from pfSense Plus to pfSense
  • Category changed from Rules / NAT to Rules / NAT
  • Target version deleted (22.11)
  • Affected Plus Version deleted (22.05)
  • Plus Target Version set to 22.11
Actions #5

Updated by Marcos M about 1 month ago

  • Related to Bug #12630: States are always created on the default gateway interface. added
Actions #6

Updated by Steve Wheeler about 1 month ago

  • Target version set to 2.7.0
  • Affected Version set to 2.6.0

Tested:
2.5.0 - Passes TCP traffic from both WANs
2.5.1 - Fails as described
2.5.2 - Fails as described
2.6.0 - Fails as described
2.7.0.a.20220812.0002 - Fails as described

Actions #7

Updated by Kristof Provost about 1 month ago

I can reproduce the problem on a 22.09 snapshot, but not on a main-based image:

[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: env LD_LIBRARY_PATH=/root ./iperf3 -t 3 -c 10.0.2.1 -B 1.0.3.89
Connecting to host 10.0.2.1, port 5201
[  5] local 1.0.3.89 port 22009 connected to 10.0.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  76.8 MBytes   644 Mbits/sec    0    325 KBytes
[  5]   1.00-2.00   sec  76.8 MBytes   644 Mbits/sec    0    325 KBytes
[  5]   2.00-3.00   sec  76.8 MBytes   645 Mbits/sec    0    325 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   230 MBytes   644 Mbits/sec    0             sender
[  5]   0.00-3.00   sec   230 MBytes   643 Mbits/sec                  receiver

iperf Done.
[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: env LD_LIBRARY_PATH=/root ./iperf3 -t 3 -c 10.0.2.1 -B 1.0.2.78
Connecting to host 10.0.2.1, port 5201
[  5] local 1.0.2.78 port 6143 connected to 10.0.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  77.5 MBytes   650 Mbits/sec    0    325 KBytes
[  5]   1.00-2.00   sec  76.8 MBytes   644 Mbits/sec    0    325 KBytes
[  5]   2.00-3.00   sec  75.8 MBytes   636 Mbits/sec    0    325 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   230 MBytes   644 Mbits/sec    0             sender
[  5]   0.00-3.00   sec   230 MBytes   643 Mbits/sec                  receiver

iperf Done.
[22.09-DEVELOPMENT][root@pfSense.home.arpa]/root: uname -a
FreeBSD pfSense.home.arpa 14.0-CURRENT FreeBSD 14.0-CURRENT #0 plus-devel-main-n255891-f8ae608637e: Tue Aug 23 11:44:07 CEST 2022     root@nut:/usr/home/kp/netgate/crossbuild-main/obj/amd64/lg6m8yhq/usr/home/kp/netgate/crossbuild-main/sources/FreeBSD-src-kp-devel-main/amd64.amd64/sys/pfSense amd64

Given that, I do not intend to investigate further. We should merely re-test this prior to release to ensure it remains fixed.

Actions

Also available in: Atom PDF