Bug #7015
closedIPsec not working behind NAT
Added by Renato Botelho over 7 years ago. Updated over 6 years ago.
0%
Description
@luiz has the details, looks like a ESP fragment but it creates odd state with unknown IP address like:
enc0 icmp 69.0.0.84:6748 <- 172.27.10.20:6748 0:0
Updated by Steve Wheeler over 7 years ago
Also seeing this after upgrading to 2.4.
Initially unable to ping across the tunnel but a packet capture showed pings leaving over IPSec and replies coming back. The replies were being blocked in the firewall, not matching the state opened. By adding a rule to pass that reply traffic I am able ping but the state created is indeed very weird:
LAN icmp 172.21.16.5:6442 -> 172.27.34.10:6442 0:0 7 / 6 588 B / 504 B
IPsec icmp 172.21.16.5:6442 -> 172.27.34.10:6442 0:0 7 / 0 588 B / 0 B
IPsec icmp 172.27.34.10:6442 -> 69.0.0.84:6442 0:0 6 / 0 504 B / 0 B
Yet the ping replies are correctly routed back to 172.21.16.5.
Also for UDP traffic:
LAN udp 172.21.16.7:5060 -> 172.27.34.10:5060 MULTIPLE:MULTIPLE 904 / 691 593 KiB / 380 KiB
IPsec udp 172.21.16.7:5060 -> 172.27.34.10:5060 SINGLE:NO_TRAFFIC 904 / 0 593 KiB / 0 B
IPsec udp 172.27.34.10:5060 -> 69.96.2.50:5060 NO_TRAFFIC:SINGLE 648 / 0 356 KiB / 0 B
That VoIP works fine though.
For TCP traffic it appears each state only sees one half of the exchange so both block the out-of-state traffic. I had to add floating rules with direction any, flags any, and sloppy states in order to pass it.
Other users on that same IPSec server hitting the same resources are not seeing this problem. My end point is behind NAT so the tunnel uses NAT-T though.
Updated by Vladimir Suhhanov over 7 years ago
Could it be related to https://redmine.pfsense.org/issues/6937 ?
Updated by Renato Botelho over 7 years ago
Vladimir Putin wrote:
Could it be related to https://redmine.pfsense.org/issues/6937 ?
I believe not. We just found out an evidence this issue only happens when WAN interface is behind NAT
Updated by Renato Botelho over 7 years ago
- Subject changed from IPsec not working with specific ISP to IPsec not working behind NAT
Updated by Luiz Souza over 7 years ago
- Status changed from Confirmed to Feedback
- % Done changed from 0 to 100
Fixed in latest update.
Updated by Luiz Souza over 7 years ago
- Status changed from Feedback to Resolved
Fixed (also confirmed by JimP on #6937 which was caused by the same issue).
Updated by David Myers over 7 years ago
I’m testing routing all IPv4 and IPv6 LAN traffic through a remote VPN server and am having issues with IPv6 that might be related to this issue.
For testing I have pfSense 2.4 Beta in a VirtualBox. The LAN interface is bridged through the Ethenet adapter with static IPv4 and IPv6 addresses. The WAN interface is NAT-ed so as to appear on a different network and only has an IPv4 address. So outgoing IPv4 traffic from this VM is NAT-ed twice, first through VirtualBox then through my real pfSense box.
I’ve set up an IKEv2 Phase 1 tunnel over IPv4, and have IPv4 and IPv6 Phase 2 tunnels. IPv4 seems to be working fine with no additional firewall rules as long as I use MSS clamping to 1400 on both sides.
IPv6 is acting strange. When I try to ping from a client system (also in VirtualBox but on a different host) to Google DNS at 2001:4860:4860::8888, it fails:
Apr 4 12:58:45 pfSense filterlog: 7,,,1000104535,enc0,match,block,in,6,0x00,0xdcb13,57,ICMPv6,58,64,2001:4860:4860::8888,2602:304:aa7e:d0a0::201,
That rule is:
block in log inet6 all tracker 1000104535 label "Default deny rule IPv6”
In the state table I see:
em1 ipv6-icmp 2001:4860:4860::8888[1319] <- 2602:304:aa7e:d0a0::201[1319] NO_TRAFFIC:NO_TRAFFIC enc0 ipv6-icmp 2602:304:aa7e:d0a0::201[1319] -> 2001:4860:4860::8888[1319] NO_TRAFFIC:NO_TRAFFIC
Running “curl ipv6.icanhazip.com” to try to determine the public IPv6 address of the remote VPN server results in several log entries like this:
Apr 4 13:07:59 pfSense filterlog: 7,,,1000104535,enc0,match,block,in,6,0x00,0x5622e,52,TCP,6,40,2604:7780:200:305:f816:3eff:fe6e:69e9,2602:304:aa7e:d0a0::201,80,38762,0,SA,1568736757,1810540167,28560,,mss;sackOK;TS;nop;wscale
These are, I think, the related states:
em1 tcp 2604:7780:200:305:f816:3eff:fe6e:69e9[80] <- 2602:304:aa7e:d0a0::201[38762] CLOSED:SYN_SENT enc0 tcp 2602:304:aa7e:d0a0::201[38762] -> 2604:7780:200:305:f816:3eff:fe6e:69e9[80] SYN_SENT:CLOSED
I would have expected this traffic to be passed by the default LAN IPv6 to any rule.
Adding the following floating rule makes the above curl command work, but not consistently:
pass on { enc0 } inet6 proto tcp from any to any tracker 1491081937 flags any keep state ( sloppy ) label “USER_RULE"
Updated by David Myers over 7 years ago
Using the setup described above I’ve also been having issues when trying to use IPsec Transport mode with either a GRE or GIF. (Perhaps what I’m seeing is really related to https://redmine.pfsense.org/issues/4479, I’m not sure).
The remote system is a Linux virtual server running strongSwan and either a GRE-to-GRE or GIF-to-IPIP tunnel. The remote inner tunnel address is 10.19.49.2. A test client on my LAN is at 192.168.1.201.
For either GRE or GIF I’m using manual (AON) NAT to avoid the automatically created outbound NAT rules for the GRE or GIF (WAN still uses NAT, though). The only rule I’m adding is on LAN to policy route the test client through the GRE or GIF gateway. So no added rules on the GIF/GRE, IPsec, WAN, or Floating tabs (and no RFC 1918 or bogons rules either).
When using a GIF, dpinger can happily ping the other side and I can see the traffic counters increment on the other side. If I try to ping from the remote VPN server to the LAN client the ping fails as expected, but apparently only if there are no other states for that client. If I cause TCP traffic from the LAN client, the ping from the remote side starts working and one of the created states is reversed:
em1 tcp 64.182.208.184:80 <- 192.168.1.201:37590 CLOSED:SYN_SENT gif0 tcp 192.168.1.201:37590 -> 64.182.208.184:80 SYN_SENT:CLOSED em1 icmp 10.19.49.2:10733 -> 192.168.1.201:10733 0:0 gif0 icmp 192.168.1.201:10733 -> 10.19.49.2:10733 0:0
Nothing appears in the firewall logs.
When using a GRE, I can’t even ping the remote side. I get the following error and the GRE interface outbound error counter increments:
PING 10.19.49.2 (10.19.49.2): 56 data bytes ping: sendto: Permission denied
If I drop IPsec the error goes away, though the pings are unanswered because there are no rules on the remote side to allow a tunnel to come in directly from the WAN.
Updated by David Myers over 7 years ago
As of 2.4.0.b.20170421.0857 I'm getting the same ping errors with a GIF. dpinger's attempts to ping do create a state:
gif0 icmp 10.19.49.1:20726 -> 10.19.49.2:20726 0:0
Updated by David Myers over 7 years ago
The problem I'm seeing with a GIF might be a command ordering or race condition issue.
Using 20170428 I set up a new IKEv2 Phase 1, Phase 2 Transport, GIF, and GIF Interface and pings were OK. I could also route LAN traffic through the GIF with a policy route. Then I rebooted and pings failed with the "Permission denied" error above.
Updated by David Myers over 7 years ago
Me again. Here's some more state strangeness.
Today the GIF tunnel is working and I can route traffic from my test client to the Internet, but file transfers stall after about 30 seconds. Here are the states when I do a wget of a large file:
em1 tcp 192.101.102.2:80 <- 192.168.1.201:40950 ESTABLISHED:ESTABLISHED gif0 tcp 192.168.1.201:40950 -> 192.101.102.2:80 SYN_SENT:CLOSED
The state on the GIF should presumably be ESTABLISHED:ESTABLISHED like the LAN state. After this state disappears the file transfer stops until wget tries again with a new connection.
Updated by Luiz Souza about 7 years ago
- Target version changed from 2.4.0 to 2.4.1
- % Done changed from 100 to 0
Updated by Wagner Sartori Junior almost 7 years ago
I just upgraded to 2.4.0-RC. My 2.3.4 was working perfectly before that, and after the upgrade immediately after the reboot, trying to ping the other side of my GRE:
$ ping -c3 172.16.0.101 PING 172.16.0.101 (172.16.0.101): 56 data bytes ping: sendto: Permission denied ping: sendto: Permission denied ping: sendto: Permission denied --- 172.16.0.101 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss
My setup is
pppoe -> ipsec -> gre <- ipsec <- datacenter
I have BGP and OSPF that's why the GRE was used.
Is there any workaround to make it able to ping? Can be a command line after reboot or anything so I don't need to reinstall with 2.3.4.
Updated by Wagner Sartori Junior almost 7 years ago
just found a workaround... reset all the states, and it starts working!
Updated by Wagner Sartori Junior almost 7 years ago
probably this is a duplicate of #4479 ?
Updated by Jim Pingle almost 7 years ago
The GRE parts may belong on that ticket, but this ticket originally did not have anything to do with GRE, someone along the way commented thinking it was related when it may not have been.
Updated by Luiz Souza almost 7 years ago
Please re-test with a recent 2.4.0-RC image based on FreeBSD 11.1.
There are significant improvements in this area in FreeBSD.
Updated by David Myers almost 7 years ago
I’m still seeing problems with IPv6 tunneling over an IPv4 IPsec connection, but the symptoms are different from what I previously reported in #7015-8 above. In particular, I no longer see traffic being blocked by firewall rules.
Small traffic like the ping and curl commands mentioned in #7015-8 seems to work properly. However when I start a large TCP transfer it eventually hangs and subsequent attempts with ping and curl then also start failing. If I wait several minutes everything starts working again until the next large transfer is attempted.
During these periods after a TCP transfer has hung, if I attempt the curl command, instead of the normal states I see that pfSense has responded with IPv6 ICMP and curl reports “No route to host”. Pings fail with “Destination unreachable: Address unreachable”.
The states related to the TCP transfer stay around indefinitely as ESTABLISHED even after the transfer has been terminated on the client.
IPv6 pings from pfSense itself seem to continue to work. It’s traffic from the client that fails.
I’m not sure I can completely rule out a problem with my test setup.
Updated by David Myers almost 7 years ago
The problem I reported above occurs when the Phase 2 IPv6 Tunnel "Local Network" is set to "LAN subnet", and therefore includes the router's LAN address. When the router's LAN address is excluded, IPv6 traffic from the client flows normally.
Edited to add:
The LAN IPv6 subnet isn't getting added to "conn bypasslan" in ipsec.conf.
Updated by Jim Pingle almost 7 years ago
- Target version changed from 2.4.1 to 2.4.2
Updated by Jim Pingle almost 7 years ago
- Target version changed from 2.4.2 to 2.4.3
Updated by Jim Pingle over 6 years ago
- Status changed from Assigned to Resolved