Project

General

Profile

Actions

Regression #15430

closed

Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on enc0

Added by Mike Moore about 2 months ago. Updated 20 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Routing
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
24.08
Release Notes:
Default
Affected Version:
Affected Architecture:

Description

https://forum.netgate.com/topic/187632/24-03-frr-has-flapping-bgp-neighbors/3

In my set up there are two VPN types where dynamic routing is occurring. Wireguard and IPsec.
IPsec has been unstable since the upgrade to 24.03.
Digging into the issue it is related to the state policy change. Adding a specific rule for BGP from neighbor to my pfsense firewall on port 179 and changing the state policy to Floating allowed BGP to remain UP without the constant flapping.

Unsure why there is a difference between the two VPN types to pf.
Redmine for tracking.

No need to troubleshoot as problem is resolved. Noting here for additional follow up if required and corner case testing by developers if deemed necessary.


Related issues

Related to Feature #11395: Option to switch IPsec filtering modes to choose between ``enc`` and ``if_ipsec`` filteringClosedJim Pingle02/10/2021

Actions
Related to Bug #8686: IPsec VTI: Assigned interface firewall rules are never parsedNew07/24/2018

Actions
Has duplicate Bug #15431: Interface Bound Firewall State Policy Breaks IPsec VTIDuplicate

Actions
Actions #1

Updated by Jim Pingle about 2 months ago

  • Project changed from pfSense Plus to pfSense
  • Category changed from Routing to Routing

What type of IPsec VPN, policy-based or VTI? Since you mention BGP, I'm guessing VTI, but it needs to be confirmed.

The if-bound state policy is much less forgiving when it comes to things like asymmetric routing, but it is possible that isn't the case and it's something in how the states are handled there.

Actions #2

Updated by Jim Pingle about 2 months ago

  • Has duplicate Bug #15431: Interface Bound Firewall State Policy Breaks IPsec VTI added
Actions #3

Updated by Mike Moore about 2 months ago

VTI mode for IPsec.

To reiterate, Wireguard VPN w/ BGP saw no issues.

Actions #4

Updated by Jim Pingle about 2 months ago

  • Subject changed from state policy changes inconsistent per VPN to Interface-bound state policy does not handle IPsec VTI traffic as expected

IPsec is fundamentally different in how it's handled compared to things like WireGuard/OpenVPN/OpenVPN+DCO. IPsec can be handled in certain ways that make the traffic appear to enter and exit on different interfaces (e.g. ipsecX vs enc0), which naturally doesn't play well with if-bound states but would pass OK with floating states.

What do you have set for the "IPsec Filter Mode" under VPN > IPsec, on the Advanced Settings tab?

Actions #5

Updated by Mike Moore about 2 months ago

IPsec Filter Mode set to 'Filter IPsec Tunnel, Transport and VTI on IPsec tab'

Actions #6

Updated by Jim Pingle about 2 months ago

  • Subject changed from Interface-bound state policy does not handle IPsec VTI traffic as expected to Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on enc0
  • Assignee set to Kristof Provost
  • Target version set to 2.8.0
  • Plus Target Version set to 24.07

If you do not have any tunnel mode IPsec (no site to site tunnel mode P2s, no mobile IPsec) you could change the filter mode to the other option and then add rules on a tab for the assigned IPsec interface(s) which would work better for this.

However if you have any policy-based/tunnel mode tunnels or mobile IPsec then you have to keep it on the setting you are using, and will likely need to use the floating state policy on IPsec rules.

I'll keep this open for the time being to see if there is anything we can do about this internally but given how IPsec works in that mode it may not be compatible with if-bound states.

Actions #7

Updated by Jim Pingle about 2 months ago

  • Related to Feature #11395: Option to switch IPsec filtering modes to choose between ``enc`` and ``if_ipsec`` filtering added
Actions #8

Updated by Jim Pingle about 2 months ago

  • Related to Bug #8686: IPsec VTI: Assigned interface firewall rules are never parsed added
Actions #9

Updated by Mike Moore about 2 months ago

I have made the firewall a VTI/Routed IPsec gateway moving forward.
Considering this drawback is noted in the documentation I don't think its a huge deal but I would mention that without putting 2 and 2 together there is no obvious way to know that the default behavior of the state policy would break FRR neighbor peering. Either the peering doesn't come up OR its flapping and its not obvious what the problem is. I am not sure whats the best way to handle that short of giving a blurb somewhere in the FRR menu or in the VPN menu.

Actions #10

Updated by Jim Pingle about 2 months ago

I added notes about this to the docs about state policy in general (and in the release notes): https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#interface-bound-states

It's not specific to FRR that just happens to be how you encountered it.

It would be great to see this addressed in the OS and/or PF if possible though so we'll keep this open for the time being unless further analysis reveals it's not feasible to fix directly.

If we can't fix it in the OS or PF we could also change the backend rule generation to use floating policy for tunneled IPsec traffic in/out, we had to make a similar workaround for that in another edge case.

Actions #11

Updated by Kris Phillips about 1 month ago

I can confirm this behavior.

Given that VTIs under the default filter mode with the default firewall rules will be configured for interface-bound rules, not floating rules, I agree we should either make IPSec rules floating intrinsically or make the firewall rules default to floating rules with the option to set them to interface-bound. This has been consistently hitting customers who have mixed VTI and policy-based tunnels.

Actions #12

Updated by Kris Phillips about 1 month ago

Ran into an issue today with inconsistency here. When trying to upload a file to a web page's PHP-based upload function, it would hang and fail, but the web page would load normally. The web page and upload destination were the same. I narrowed this down to interface-bound rules passing traffic over an IPSec tunnel. Once I switched to one of the following, the issue went away:

1. Switched the outbound policy-based route to the VPN tunnel to Floating from the inside interface
2. Switched to VTI-only filter mode for IPSec

Actions #13

Updated by Marcos M 26 days ago

  • File 15430_plus.txt added
  • Status changed from New to Pull Request Review
  • Assignee changed from Kristof Provost to Marcos M

We can try to work around the issue until #8686 is resolved.
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/1153

A patch is attached for testing. To test it, apply the patch with the System Patches package, then reload the filter (Status > Filter Reload).

Actions #14

Updated by Jim Pingle 22 days ago

  • Plus Target Version changed from 24.07 to 24.08
Actions #15

Updated by Marcos M 22 days ago

  • Status changed from Pull Request Review to Feedback
  • % Done changed from 0 to 100
Actions #16

Updated by Marcos M 21 days ago

  • File deleted (15430_plus.txt)
Actions #17

Updated by Mike Moore 20 days ago

Patch applied.
Should i undo my previous changes of floating policy?

Actions #18

Updated by Marcos M 20 days ago

There was an additional change after that, use the following instead; this should hopefully be included in the System Patches package soon: Show

And yes, previous rule changes may be reverted.

Actions #19

Updated by Mike Moore 20 days ago

For validation i see my bgp peers haven't dropped.

Actions #20

Updated by Marcos M 20 days ago

  • Status changed from Feedback to Resolved

Great, thanks for confirming!

Actions

Also available in: Atom PDF