Regression #15430
closedInterface-bound state policy does not handle IPsec VTI traffic as expected when filtering on ``enc0`` interface
Added by Mike Moore 7 months ago. Updated about 1 month ago.
100%
Description
https://forum.netgate.com/topic/187632/24-03-frr-has-flapping-bgp-neighbors/3
In my set up there are two VPN types where dynamic routing is occurring. Wireguard and IPsec.
IPsec has been unstable since the upgrade to 24.03.
Digging into the issue it is related to the state policy change. Adding a specific rule for BGP from neighbor to my pfsense firewall on port 179 and changing the state policy to Floating allowed BGP to remain UP without the constant flapping.
Unsure why there is a difference between the two VPN types to pf.
Redmine for tracking.
No need to troubleshoot as problem is resolved. Noting here for additional follow up if required and corner case testing by developers if deemed necessary.
Related issues
Updated by Jim Pingle 7 months ago
- Project changed from pfSense Plus to pfSense
- Category changed from Routing to Routing
What type of IPsec VPN, policy-based or VTI? Since you mention BGP, I'm guessing VTI, but it needs to be confirmed.
The if-bound state policy is much less forgiving when it comes to things like asymmetric routing, but it is possible that isn't the case and it's something in how the states are handled there.
Updated by Jim Pingle 7 months ago
- Has duplicate Bug #15431: Interface Bound Firewall State Policy Breaks IPsec VTI added
Updated by Mike Moore 7 months ago
VTI mode for IPsec.
To reiterate, Wireguard VPN w/ BGP saw no issues.
Updated by Jim Pingle 7 months ago
- Subject changed from state policy changes inconsistent per VPN to Interface-bound state policy does not handle IPsec VTI traffic as expected
IPsec is fundamentally different in how it's handled compared to things like WireGuard/OpenVPN/OpenVPN+DCO. IPsec can be handled in certain ways that make the traffic appear to enter and exit on different interfaces (e.g. ipsecX
vs enc0
), which naturally doesn't play well with if-bound states but would pass OK with floating states.
What do you have set for the "IPsec Filter Mode" under VPN > IPsec, on the Advanced Settings tab?
Updated by Mike Moore 7 months ago
IPsec Filter Mode set to 'Filter IPsec Tunnel, Transport and VTI on IPsec tab'
Updated by Jim Pingle 7 months ago
- Subject changed from Interface-bound state policy does not handle IPsec VTI traffic as expected to Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on enc0
- Assignee set to Kristof Provost
- Target version set to 2.8.0
- Plus Target Version set to 24.07
If you do not have any tunnel mode IPsec (no site to site tunnel mode P2s, no mobile IPsec) you could change the filter mode to the other option and then add rules on a tab for the assigned IPsec interface(s) which would work better for this.
However if you have any policy-based/tunnel mode tunnels or mobile IPsec then you have to keep it on the setting you are using, and will likely need to use the floating state policy on IPsec rules.
I'll keep this open for the time being to see if there is anything we can do about this internally but given how IPsec works in that mode it may not be compatible with if-bound states.
Updated by Jim Pingle 7 months ago
- Related to Feature #11395: Option to switch IPsec filtering modes to choose between ``enc`` and ``if_ipsec`` filtering added
Updated by Jim Pingle 7 months ago
- Related to Bug #8686: IPsec VTI: Assigned interface firewall rules are never parsed added
Updated by Mike Moore 7 months ago
I have made the firewall a VTI/Routed IPsec gateway moving forward.
Considering this drawback is noted in the documentation I don't think its a huge deal but I would mention that without putting 2 and 2 together there is no obvious way to know that the default behavior of the state policy would break FRR neighbor peering. Either the peering doesn't come up OR its flapping and its not obvious what the problem is. I am not sure whats the best way to handle that short of giving a blurb somewhere in the FRR menu or in the VPN menu.
Updated by Jim Pingle 7 months ago
I added notes about this to the docs about state policy in general (and in the release notes): https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#interface-bound-states
It's not specific to FRR that just happens to be how you encountered it.
It would be great to see this addressed in the OS and/or PF if possible though so we'll keep this open for the time being unless further analysis reveals it's not feasible to fix directly.
If we can't fix it in the OS or PF we could also change the backend rule generation to use floating policy for tunneled IPsec traffic in/out, we had to make a similar workaround for that in another edge case.
Updated by Kris Phillips 6 months ago
I can confirm this behavior.
Given that VTIs under the default filter mode with the default firewall rules will be configured for interface-bound rules, not floating rules, I agree we should either make IPSec rules floating intrinsically or make the firewall rules default to floating rules with the option to set them to interface-bound. This has been consistently hitting customers who have mixed VTI and policy-based tunnels.
Updated by Kris Phillips 6 months ago
Ran into an issue today with inconsistency here. When trying to upload a file to a web page's PHP-based upload function, it would hang and fail, but the web page would load normally. The web page and upload destination were the same. I narrowed this down to interface-bound rules passing traffic over an IPSec tunnel. Once I switched to one of the following, the issue went away:
1. Switched the outbound policy-based route to the VPN tunnel to Floating from the inside interface
2. Switched to VTI-only filter mode for IPSec
Updated by Marcos M 6 months ago
- File 15430_plus.txt added
- Status changed from New to Pull Request Review
- Assignee changed from Kristof Provost to Marcos M
We can try to work around the issue until #8686 is resolved.
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/1153
A patch is attached for testing. To test it, apply the patch with the System Patches package, then reload the filter (Status > Filter Reload).
Updated by Jim Pingle 6 months ago
- Plus Target Version changed from 24.07 to 24.08
Updated by Marcos M 6 months ago
- Status changed from Pull Request Review to Feedback
- % Done changed from 0 to 100
Applied in changeset e254aea45c3694ff280247be7670421b86d5bb31.
Updated by Mike Moore 6 months ago
Patch applied.
Should i undo my previous changes of floating policy?
Updated by Marcos M 6 months ago
There was an additional change after that, use the following instead; this should hopefully be included in the System Patches package soon: ShowHide
diff --git a/src/etc/inc/filter.inc b/src/etc/inc/filter.inc index 9ebbfd5678..6fa0d7e177 100644 --- a/src/etc/inc/filter.inc +++ b/src/etc/inc/filter.inc @@ -174,6 +174,31 @@ if (config_get_path('ipsec/filtermode') == 'if_ipsec') { $filter_interface_remove[] = 'ipsec'; } +/** + * Floating states for IPsec rules are not needed when: + * - The global state policy is already set to floating + * - The IPsec Filter Mode is set to assigned interfaces + * - There are no enabled VTI IPsec tunnels + */ +global $use_floating_states_for_ipsec; +$use_floating_states_for_ipsec = false; +if (!config_path_enabled('system', 'disable_auto_floating_states') && + (config_get_path('system/statepolicy', 'if-bound') == 'if-bound') && + (config_get_path('ipsec/filtermode', 'enc') == 'enc')) { + // look for a VTI or transport mode p2 + foreach (config_get_path('ipsec/phase2', []) as $ipsecp2) { + if (!isset($ipsecp2['disabled']) && ($ipsecp2['mode'] == 'transport') || ($ipsecp2['mode'] == 'vti')) { + // check if the matching p1 to this p2 is enabled + foreach (config_get_path('ipsec/phase1', []) as $ipsecp1) { + if (!isset($ipsecp1['disabled']) && ($ipsecp1['ikeid'] == $ipsecp2['ikeid'])) { + $use_floating_states_for_ipsec = true; + break 2; + } + } + } + } +} + /* * Fixed tracker values (used to group and track usage in GUI): * @@ -3356,7 +3381,7 @@ function filter_generate_address(& $rule, $target = "source", $isnat = false) { } function filter_generate_user_rule($rule, & $extralabels = null) { - global $g, $FilterIflist, $GatewaysList, $vpns_list; + global $g, $FilterIflist, $GatewaysList, $vpns_list, $use_floating_states_for_ipsec; global $dummynet_name_list, $vlanprio_values, $time_based_rules; if (config_path_enabled('system','developerspew')) { @@ -3692,10 +3717,14 @@ function filter_generate_user_rule($rule, & $extralabels = null) { (isset($rule['statepolicy']) and !empty($rule['statepolicy'])) or (isset($rule['sloppy'])) or ((config_path_enabled('pflow') and ($rule['pflow'] != 'disabled')) and (config_path_enabled('pflow', 'default') or ($rule['pflow'] == 'enabled'))) or - (isset($rule['nopfsync']))) { + (isset($rule['nopfsync'])) or + ($use_floating_states_for_ipsec && ($rule['interface'] == 'enc0'))) { $aline['flags'] .= "( "; if (!empty($rule['statepolicy'])) { $aline['flags'] .= "{$rule['statepolicy']} "; + } elseif ($use_floating_states_for_ipsec && ($rule['interface'] == 'enc0')) { + // a floating state is needed for enc0 rules - see #15430 + $aline['flags'] .= 'floating '; } if ((config_path_enabled('pflow') && ($rule['pflow'] != 'disabled')) && @@ -4094,7 +4123,7 @@ function filter_eth_rules_generate(bool $use_cache = true): string { function filter_rules_generate() { global $FilterIflist, $time_based_rules, $GatewaysList, - $tracker, $vlanprio_values, $vpn_and_ppp_ifs; + $tracker, $vlanprio_values, $vpn_and_ppp_ifs, $use_floating_states_for_ipsec; global $eth_rules_cache; @@ -4577,7 +4606,9 @@ EOD; require_once("ipsec.inc"); } if (ipsec_enabled()) { - $ipfrules .= "pass out {$log['pass']} on \$IPsec all ridentifier {$increment_tracker()} ridentifier {$increment_tracker()} keep state label \"IPsec internal host to host\"\n"; + // a floating state is needed for ipsec rules - see #15430 + $state_policy = $use_floating_states_for_ipsec ? '(floating)' : ''; + $ipfrules .= "pass out {$log['pass']} on \$IPsec all ridentifier {$increment_tracker()} ridentifier {$increment_tracker()} keep state {$state_policy} label \"IPsec internal host to host\"\n"; } $saved_tracker += 10; diff --git a/src/usr/local/pfSense/include/www/system_advanced_firewall.inc b/src/usr/local/pfSense/include/www/system_advanced_firewall.inc index 5484323d4d..e5a0e83ccf 100644 --- a/src/usr/local/pfSense/include/www/system_advanced_firewall.inc +++ b/src/usr/local/pfSense/include/www/system_advanced_firewall.inc @@ -46,6 +46,7 @@ function getSystemAdvancedFirewall($json = false) { $pconfig['bypassstaticroutes'] = isset($config['filter']['bypassstaticroutes']); $pconfig['checkaliasesurlcert'] = isset($config['system']['checkaliasesurlcert']); $pconfig['statepolicy'] = config_get_path('system/statepolicy'); + $pconfig['disable_auto_floating_states'] = config_get_path('system/disable_auto_floating_states'); $pconfig['disablefilter'] = config_get_path('system/disablefilter'); $pconfig['enableethfilter'] = config_path_enabled('system', 'enableethfilter'); $pconfig['disablenatreflection'] = config_get_path('system/disablenatreflection'); @@ -207,6 +208,11 @@ function saveSystemAdvancedFirewall($post, $json = false) { } config_set_path('system/statepolicy', $post['statepolicy']); + if (isset($post['disable_auto_floating_states'])) { + config_set_path('system/disable_auto_floating_states', true); + } else { + config_del_path('system/disable_auto_floating_states'); + } if ($post['enableethfilter'] === 'yes') { config_set_path('system/enableethfilter', true); diff --git a/src/usr/local/www/system_advanced_firewall.php b/src/usr/local/www/system_advanced_firewall.php index ab9ea187ab..f92f68c34e 100644 --- a/src/usr/local/www/system_advanced_firewall.php +++ b/src/usr/local/www/system_advanced_firewall.php @@ -277,6 +277,18 @@ $section->addInput(new Form_Select( 'between either mode.', '</span><ul class="help-block"><li>', '</li><li>', '</li></ul><span class="help-block">'); +$section->addInput(new Form_Checkbox( + 'disable_auto_floating_states', + 'Disable state-policy override for IPsec rules', + 'Don\'t automatically use floating states for IPsec rules.', + isset($pconfig['disable_auto_floating_states']) +))->setHelp('When %1$sFirewall State Policy%2$s is set to %1$sInterface '. + 'Bound States%2$s, unchecking this option allows IPsec rules to '. + 'automatically use floating states where needed. This option is '. + 'ignored when %3$sIPsec Filter Mode%4$s is set to assigned '. + 'interfaces.', '<b>', '</b>','<a href="firewall_nat_out.php">', '</a>' +); + $section->addInput(new Form_Checkbox( 'enableethfilter', 'Ethernet Filtering',
And yes, previous rule changes may be reverted.
Updated by Mike Moore 6 months ago
For validation i see my bgp peers haven't dropped.
Updated by Jim Pingle about 2 months ago
- Subject changed from Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on enc0 to Interface-bound state policy does not handle IPsec VTI traffic as expected when filtering on ``enc0`` interface
Updated by Jim Pingle about 1 month ago
- Plus Target Version changed from 24.08 to 24.11