pfSense

Also seeing this after upgrading to 2.4.

Initially unable to ping across the tunnel but a packet capture showed pings leaving over IPSec and replies coming back. The replies were being blocked in the firewall, not matching the state opened. By adding a rule to pass that reply traffic I am able ping but the state created is indeed very weird:

LAN icmp 172.21.16.5:6442 -> 172.27.34.10:6442 0:0 7 / 6 588 B / 504 B
IPsec icmp 172.21.16.5:6442 -> 172.27.34.10:6442 0:0 7 / 0 588 B / 0 B
IPsec icmp 172.27.34.10:6442 -> 69.0.0.84:6442 0:0 6 / 0 504 B / 0 B

Yet the ping replies are correctly routed back to 172.21.16.5.

Also for UDP traffic:

LAN udp 172.21.16.7:5060 -> 172.27.34.10:5060 MULTIPLE:MULTIPLE 904 / 691 593 KiB / 380 KiB
IPsec udp 172.21.16.7:5060 -> 172.27.34.10:5060 SINGLE:NO_TRAFFIC 904 / 0 593 KiB / 0 B
IPsec udp 172.27.34.10:5060 -> 69.96.2.50:5060 NO_TRAFFIC:SINGLE 648 / 0 356 KiB / 0 B

That VoIP works fine though.

For TCP traffic it appears each state only sees one half of the exchange so both block the out-of-state traffic. I had to add floating rules with direction any, flags any, and sloppy states in order to pass it.

Other users on that same IPSec server hitting the same resources are not seeing this problem. My end point is behind NAT so the tunnel uses NAT-T though.

Could it be related to https://redmine.pfsense.org/issues/6937 ?

Vladimir Putin wrote:

Could it be related to https://redmine.pfsense.org/issues/6937 ?

I believe not. We just found out an evidence this issue only happens when WAN interface is behind NAT

I’m testing routing all IPv4 and IPv6 LAN traffic through a remote VPN server and am having issues with IPv6 that might be related to this issue.

For testing I have pfSense 2.4 Beta in a VirtualBox. The LAN interface is bridged through the Ethenet adapter with static IPv4 and IPv6 addresses. The WAN interface is NAT-ed so as to appear on a different network and only has an IPv4 address. So outgoing IPv4 traffic from this VM is NAT-ed twice, first through VirtualBox then through my real pfSense box.

I’ve set up an IKEv2 Phase 1 tunnel over IPv4, and have IPv4 and IPv6 Phase 2 tunnels. IPv4 seems to be working fine with no additional firewall rules as long as I use MSS clamping to 1400 on both sides.

IPv6 is acting strange. When I try to ping from a client system (also in VirtualBox but on a different host) to Google DNS at 2001:4860:4860::8888, it fails:

Apr  4 12:58:45 pfSense filterlog: 7,,,1000104535,enc0,match,block,in,6,0x00,0xdcb13,57,ICMPv6,58,64,2001:4860:4860::8888,2602:304:aa7e:d0a0::201,

block in log inet6 all tracker 1000104535 label "Default deny rule IPv6”

em1 ipv6-icmp 2001:4860:4860::8888[1319] <- 2602:304:aa7e:d0a0::201[1319]       NO_TRAFFIC:NO_TRAFFIC
enc0 ipv6-icmp 2602:304:aa7e:d0a0::201[1319] -> 2001:4860:4860::8888[1319]       NO_TRAFFIC:NO_TRAFFIC

Apr  4 13:07:59 pfSense filterlog: 7,,,1000104535,enc0,match,block,in,6,0x00,0x5622e,52,TCP,6,40,2604:7780:200:305:f816:3eff:fe6e:69e9,2602:304:aa7e:d0a0::201,80,38762,0,SA,1568736757,1810540167,28560,,mss;sackOK;TS;nop;wscale

These are, I think, the related states:

em1 tcp 2604:7780:200:305:f816:3eff:fe6e:69e9[80] <- 2602:304:aa7e:d0a0::201[38762]       CLOSED:SYN_SENT
enc0 tcp 2602:304:aa7e:d0a0::201[38762] -> 2604:7780:200:305:f816:3eff:fe6e:69e9[80]       SYN_SENT:CLOSED

I would have expected this traffic to be passed by the default LAN IPv6 to any rule.

Adding the following floating rule makes the above curl command work, but not consistently:

pass  on {  enc0  } inet6 proto tcp  from any to any tracker 1491081937 flags any keep state ( sloppy  )  label “USER_RULE" 

Using the setup described above I’ve also been having issues when trying to use IPsec Transport mode with either a GRE or GIF. (Perhaps what I’m seeing is really related to https://redmine.pfsense.org/issues/4479, I’m not sure).

The remote system is a Linux virtual server running strongSwan and either a GRE-to-GRE or GIF-to-IPIP tunnel. The remote inner tunnel address is 10.19.49.2. A test client on my LAN is at 192.168.1.201.

For either GRE or GIF I’m using manual (AON) NAT to avoid the automatically created outbound NAT rules for the GRE or GIF (WAN still uses NAT, though). The only rule I’m adding is on LAN to policy route the test client through the GRE or GIF gateway. So no added rules on the GIF/GRE, IPsec, WAN, or Floating tabs (and no RFC 1918 or bogons rules either).

When using a GIF, dpinger can happily ping the other side and I can see the traffic counters increment on the other side. If I try to ping from the remote VPN server to the LAN client the ping fails as expected, but apparently only if there are no other states for that client. If I cause TCP traffic from the LAN client, the ping from the remote side starts working and one of the created states is reversed:

em1 tcp 64.182.208.184:80 <- 192.168.1.201:37590       CLOSED:SYN_SENT
gif0 tcp 192.168.1.201:37590 -> 64.182.208.184:80       SYN_SENT:CLOSED
em1 icmp 10.19.49.2:10733 -> 192.168.1.201:10733       0:0
gif0 icmp 192.168.1.201:10733 -> 10.19.49.2:10733       0:0

Nothing appears in the firewall logs.

When using a GRE, I can’t even ping the remote side. I get the following error and the GRE interface outbound error counter increments:

PING 10.19.49.2 (10.19.49.2): 56 data bytes
ping: sendto: Permission denied

If I drop IPsec the error goes away, though the pings are unanswered because there are no rules on the remote side to allow a tunnel to come in directly from the WAN.

As of 2.4.0.b.20170421.0857 I'm getting the same ping errors with a GIF. dpinger's attempts to ping do create a state:

gif0 icmp 10.19.49.1:20726 -> 10.19.49.2:20726       0:0

The problem I'm seeing with a GIF might be a command ordering or race condition issue.

Using 20170428 I set up a new IKEv2 Phase 1, Phase 2 Transport, GIF, and GIF Interface and pings were OK. I could also route LAN traffic through the GIF with a policy route. Then I rebooted and pings failed with the "Permission denied" error above.

Me again. Here's some more state strangeness.

Today the GIF tunnel is working and I can route traffic from my test client to the Internet, but file transfers stall after about 30 seconds. Here are the states when I do a wget of a large file:

em1 tcp 192.101.102.2:80 <- 192.168.1.201:40950       ESTABLISHED:ESTABLISHED
gif0 tcp 192.168.1.201:40950 -> 192.101.102.2:80       SYN_SENT:CLOSED

The state on the GIF should presumably be ESTABLISHED:ESTABLISHED like the LAN state. After this state disappears the file transfer stops until wget tries again with a new connection.

I just upgraded to 2.4.0-RC. My 2.3.4 was working perfectly before that, and after the upgrade immediately after the reboot, trying to ping the other side of my GRE:

$ ping -c3 172.16.0.101
PING 172.16.0.101 (172.16.0.101): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
--- 172.16.0.101 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

My setup is

pppoe -> ipsec -> gre <- ipsec <- datacenter

I have BGP and OSPF that's why the GRE was used.

Is there any workaround to make it able to ping? Can be a command line after reboot or anything so I don't need to reinstall with 2.3.4.

just found a workaround... reset all the states, and it starts working!

probably this is a duplicate of #4479 ?

The GRE parts may belong on that ticket, but this ticket originally did not have anything to do with GRE, someone along the way commented thinking it was related when it may not have been.

Please re-test with a recent 2.4.0-RC image based on FreeBSD 11.1.

There are significant improvements in this area in FreeBSD.

I’m still seeing problems with IPv6 tunneling over an IPv4 IPsec connection, but the symptoms are different from what I previously reported in #7015-8 above. In particular, I no longer see traffic being blocked by firewall rules.

Small traffic like the ping and curl commands mentioned in #7015-8 seems to work properly. However when I start a large TCP transfer it eventually hangs and subsequent attempts with ping and curl then also start failing. If I wait several minutes everything starts working again until the next large transfer is attempted.

During these periods after a TCP transfer has hung, if I attempt the curl command, instead of the normal states I see that pfSense has responded with IPv6 ICMP and curl reports “No route to host”. Pings fail with “Destination unreachable: Address unreachable”.

The states related to the TCP transfer stay around indefinitely as ESTABLISHED even after the transfer has been terminated on the client.

IPv6 pings from pfSense itself seem to continue to work. It’s traffic from the client that fails.

I’m not sure I can completely rule out a problem with my test setup.

The problem I reported above occurs when the Phase 2 IPv6 Tunnel "Local Network" is set to "LAN subnet", and therefore includes the router's LAN address. When the router's LAN address is excluded, IPv6 traffic from the client flows normally.

Edited to add:

The LAN IPv6 subnet isn't getting added to "conn bypasslan" in ipsec.conf.