Regression #16354
closedRouting issue with BGP MultiPath Site-to-site VPN, only using OpenVPN and with the new "Interface Bound States" state policy
0%
Description
Recently upgraded from pfSense CE 2.7.2 to 2.8.0.
All seems fine, except for some problems with multipath BGP routing over VPN using OpenVPN site-to-site tunnels.
The problem seems to be due to the change in Firewall State Policy mode that occurred with the release change (and also documented in the release notes).
However, with the same setup, the problem only occurs on OpenVPN tunnels, not on IPSec (probably because of the available option "Disable state-policy override for IPsec rules" ...? )
Test setup and details.
Config¶
Site A - Configuration¶
Dual WAN (2x LTE connections)
LAN: 172.20.0.0/25
Tunnel 1 (WAN 1): 172.19.101.2/30
Tunnel 2 (WAN 2): 172.19.102.2/30
FRR BGP with multipath enabled
OpenVPN Client role
Site A - Routing Table:¶
Destination Gateway Flags Netif Expire
default 192.168.8.1 UGS ix1
172.19.101.0/30 link#13 U ovpnc1
172.19.102.0/30 link#14 U ovpnc2
172.20.0.0/25 link#1 U ix0
172.30.0.0/25 172.19.102.1 UG1 ovpnc2
172.30.0.0/25 172.19.101.1 UG1 ovpnc1
...
Site B - Configuration¶
Single WAN (FTTH)
LAN: 172.30.0.0/25
Tunnel 1: 172.19.101.1/30
Tunnel 2: 172.19.102.1/30
BGP with multipath enabled
OpenVPN Server role
Site B - Routing Table:¶
Destination Gateway Flags Netif Expire
0.0.0.0 192.168.1.254 UGS igb4
172.19.101.0/30 link#15 U ovpns2
172.19.102.0/30 link#16 U ovpns4
172.20.0.0/25 172.19.101.2 UG1 ovpns2
172.20.0.0/25 172.19.102.2 UG1 ovpns4
...
Problem Description¶
After upgrading from pfSense 2.7.2 to 2.8.0, inter-site connectivity became extremely unstable (almost always broken).
The issue manifests as:
SSH connections hang after initial handshake - Connections appear to establish but then freeze
Firewall logs show a lot of blocked packets (marked with the arrow and "direction is out") on the site that is receiving the connection
State table shows inconsistent states across interfaces
Root Cause¶
The issue appears to be caused by asymmetric routing where:
Outbound traffic uses one VPN tunnel (selected by BGP)
Return traffic uses a different tunnel (also selected by BGP independently)
pfSense 2.8.0's stricter state tracking sees this as invalid traffic
Validation¶
When disabling either VPN tunnel (leaving only one active), the problem completely disappears, confirming this is an asymmetric routing issue.
Attempted Solutions¶
Verified firewall rules - All necessary allow rules are in place
Checked VPN tunnel stability - Both tunnels are stable (confirmed also with iperf3)
Analyzed routing tables - BGP multipath is working as expected
Changing the State Policy to "Floating" nor State type to Sloopy, on a per-rule basis , seems to have no effect
Workaround¶
Changing the Firewall State Policy (System > Advanced > Firewall & NAT) from "Interface Bound States" to " Floating States " completely resolves the issue.
Additional Findings¶
OpenVPN vs IPSec VTI
This issue appears to be specific to OpenVPN tunnels.
I have another site-to-site connection with identical BGP multipath configuration using IPSec VTI, and it works perfectly without any issues in pfSense 2.8.0.
Could it be that only traffic on IPSec tunnels is managed differently?
In particular, as indicated at the beginning, I noticed that there is the option "Disable state-policy override for IPsec rules" (When Firewall State Policy is set to Interface Bound States, unchecking this option allows IPsec rules to automatically use floating states where needed. This option is ignored when IPsec Filter Mode is set to assigned interfaces.).
Obviously in my case it is not selected (default option)
This could explain the difference in behavior between IPSec and OpenVPN.
Already discussed on Netgate forum:
https://forum.netgate.com/topic/197821/routing-issue-with-multi-path-vpn-after-upgrading-to-pfsense-2-8-0
Updated by Jim Pingle 8 days ago
- Status changed from New to Not a Bug
This is not a bug, it's working like it should for OpenVPN and the "workaround" is exactly what should be done in this scenario.
It happens to work on IPsec because of how IPsec works with VTI interfaces by default since it automatically gets set to floating states, but as you noted, you need to change the state policy if you want to use multipath routing in that way on anything else.