pfSense

Re-tested this since we have a new base OS on 2.5.0. Unfortunately, this still behaves the same way on 12.1-STABLE:

Adding a pf rule on a VTI interface, it does not function as expected. With no rules on enc0, and a pass all rule on ipsec4000, no traffic is passed.

@145(1528159274) pass in quick on ipsec4000 reply-to (ipsec4000 10.6.106.2) inet all flags S/SA keep state label "USER_RULE: VTI Test Rule" 
  [ Evaluations: 16050     Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: pid 21034 State Creations: 0     ]

Note that the evaluations counter increases, but it never matches.

tcpdump does not show any packets arriving on the ipsec4000 interface with this rule present.

If you add a rule to enc0 to pass the traffic, it works and then traffic also appears in tcpdump captures on ipsec4000 as it flows.

Packets are always visible in tcpdump on enc0.

Is this related:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232522

filtertunnel sysctls seem to be 0 in pfsense.

That is certainly worth testing but we've had problems flipping that in the past (See #2993, #2636, and several forum threads) so it may not be a general solution.

If it works, we can document it at least. But we'd also have to check for regressions in behavior of tunneled (non-VTI) IPsec, transport mode IPsec, and general IPsec throughput/performance.

It doesn't appear to be related. Setting that sysctl to 1, the traffic still arrives on enc0 and is blocked by pf inbound on enc0.

I thought it was noted here but I don't see it. There is another FreeBSD issue at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474 in which they recommend disabling pfil for if_enc and setting filtertunnel similar to the above:

net.enc.out.ipsec_filter_mask=0
net.enc.in.ipsec_filter_mask=0
net.inet.ipsec.filtertunnel=1
net.inet6.ipsec6.filtertunnel=1

However, that cripples policy-based tunnels since there would now be no way to filter their traffic.

There is still no acceptable solution that allows both to work at once.

In addition to the above, the BPF mask also needs changed.

The complete set of required sysctl values are:

sysctl net.inet.ipsec.filtertunnel=1
sysctl net.inet6.ipsec6.filtertunnel=1
sysctl net.enc.out.ipsec_bpf_mask=0
sysctl net.enc.in.ipsec_bpf_mask=0
sysctl net.enc.out.ipsec_filter_mask=0
sysctl net.enc.in.ipsec_filter_mask=0

And then change source:src/etc/inc/filter.inc#L101 to:

$filter_interface_blacklist = array();

With that in place, rules on assigned VTI interface tabs are respected and rules on the IPsec tab (enc0) are ignored. Since it's default deny, no traffic on enc0 will pass. (Meaning that policy-based tunnels cannot pass traffic no matter what, only assigned VTI w/rules on tabs can)

I'm moving the option I mentioned above to a separate issue: #11395

This can remain open for the longer term question of whether or not it is possible in FreeBSD to have both behaviors at the same time, rather than forcing the user to choose.

Could the ipsec interface be enabled for inclusion to an interface group when the advanced ipsec filter mode is set to "Filter IPsec VTI and Transport on assigned interfaces, block all tunnel mode traffic"?

The reason would be to easily apply all filter rules for a typical hub-spoke setup.