Regression #15430
open
Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on enc0
Added by Mike Moore 25 days ago.
Updated 7 days ago.
Plus Target Version:
24.07
Description
https://forum.netgate.com/topic/187632/24-03-frr-has-flapping-bgp-neighbors/3
In my set up there are two VPN types where dynamic routing is occurring. Wireguard and IPsec.
IPsec has been unstable since the upgrade to 24.03.
Digging into the issue it is related to the state policy change. Adding a specific rule for BGP from neighbor to my pfsense firewall on port 179 and changing the state policy to Floating allowed BGP to remain UP without the constant flapping.
Unsure why there is a difference between the two VPN types to pf.
Redmine for tracking.
No need to troubleshoot as problem is resolved. Noting here for additional follow up if required and corner case testing by developers if deemed necessary.
- Project changed from pfSense Plus to pfSense
- Category changed from Routing to Routing
What type of IPsec VPN, policy-based or VTI? Since you mention BGP, I'm guessing VTI, but it needs to be confirmed.
The if-bound state policy is much less forgiving when it comes to things like asymmetric routing, but it is possible that isn't the case and it's something in how the states are handled there.
- Has duplicate Bug #15431: Interface Bound Firewall State Policy Breaks IPsec VTI added
VTI mode for IPsec.
To reiterate, Wireguard VPN w/ BGP saw no issues.
- Subject changed from state policy changes inconsistent per VPN to Interface-bound state policy does not handle IPsec VTI traffic as expected
IPsec is fundamentally different in how it's handled compared to things like WireGuard/OpenVPN/OpenVPN+DCO. IPsec can be handled in certain ways that make the traffic appear to enter and exit on different interfaces (e.g. ipsecX vs enc0), which naturally doesn't play well with if-bound states but would pass OK with floating states.
What do you have set for the "IPsec Filter Mode" under VPN > IPsec, on the Advanced Settings tab?
IPsec Filter Mode set to 'Filter IPsec Tunnel, Transport and VTI on IPsec tab'
- Subject changed from Interface-bound state policy does not handle IPsec VTI traffic as expected to Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on enc0
- Assignee set to Kristof Provost
- Target version set to 2.8.0
- Plus Target Version set to 24.07
If you do not have any tunnel mode IPsec (no site to site tunnel mode P2s, no mobile IPsec) you could change the filter mode to the other option and then add rules on a tab for the assigned IPsec interface(s) which would work better for this.
However if you have any policy-based/tunnel mode tunnels or mobile IPsec then you have to keep it on the setting you are using, and will likely need to use the floating state policy on IPsec rules.
I'll keep this open for the time being to see if there is anything we can do about this internally but given how IPsec works in that mode it may not be compatible with if-bound states.
- Related to Feature #11395: Option to switch IPsec filtering modes to choose between ``enc`` and ``if_ipsec`` filtering added
- Related to Bug #8686: IPsec VTI: Assigned interface firewall rules are never parsed added
I have made the firewall a VTI/Routed IPsec gateway moving forward.
Considering this drawback is noted in the documentation I don't think its a huge deal but I would mention that without putting 2 and 2 together there is no obvious way to know that the default behavior of the state policy would break FRR neighbor peering. Either the peering doesn't come up OR its flapping and its not obvious what the problem is. I am not sure whats the best way to handle that short of giving a blurb somewhere in the FRR menu or in the VPN menu.
I added notes about this to the docs about state policy in general (and in the release notes): https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#interface-bound-states
It's not specific to FRR that just happens to be how you encountered it.
It would be great to see this addressed in the OS and/or PF if possible though so we'll keep this open for the time being unless further analysis reveals it's not feasible to fix directly.
If we can't fix it in the OS or PF we could also change the backend rule generation to use floating policy for tunneled IPsec traffic in/out, we had to make a similar workaround for that in another edge case.
I can confirm this behavior.
Given that VTIs under the default filter mode with the default firewall rules will be configured for interface-bound rules, not floating rules, I agree we should either make IPSec rules floating intrinsically or make the firewall rules default to floating rules with the option to set them to interface-bound. This has been consistently hitting customers who have mixed VTI and policy-based tunnels.
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by