IPsec in transport mode, tunneled traffic does not flow through enc0
When using IPsec in transport mode, the arriving packets show as arriving encrypted on the physical interface, but the packets never show up on enc0. However, hey are actually received and processed by the OS, as reply packets are issued and are seen leaving enc0.
This happens on 2.0.x and 2.1.
Because of the way the traffic arrives, the packets never hit pf, so they don't get a state in the inbound direction. Due to that, the usual asymmetric state issues occur: ICMP works once a reply makes a state, same for UDP and other stateless protocols, but TCP is broken unless you manually add a floating rule to pass out traffic with any TCP flags and use sloppy state.
Test VMs are currently setup and available with tunnels to test/debug.
Updated by Jim Pingle over 8 years ago
OK, so net.inet.ipsec.filtertunnel=1 does get the traffic to pf, but it does not get it into tcpdump.
To see the traffic in tcpdump, you also need to set net.enc.in.ipsec_bpf_mask=1
Either of those changes may negatively impact the processing of tunneled (not transport mode) traffic, but in cases where there are only transport mode entries being used, it will help.
Updated by Per von Zweigbergk over 8 years ago
Just to note what I have discovered:
When using transport mode, setting the net.inet.ipsec.filtertunnel=1 sysctl does make it possible to filter the inner IPsec traffic and create states based upon it. However, the ruleset being matched is the ruleset on the interface on which the encrypted arrived on, NOT on the IPsec tab! This makes sense if you think about it, but might be a bit unexpected.