Feature #9545
closed
Enable Multipath Routing in the Kernel
Added by Jim Pingle almost 5 years ago. Updated 9 months ago.
100%
Description
Now that ROUTE_MPATH is in the default kernel on FreeBSD 14 and net.route.multipath is on (1), enable the MULTIPATH option in FRR.
Related issues
Updated by Jim Pingle over 4 years ago
- Status changed from New to Feedback
- % Done changed from 0 to 100
Applied in changeset pfsense:1836b0c237efdf9bf2ce9fab798f2718f0fd6028.
Updated by Jim Pingle over 4 years ago
There have been reports of instability with some routing scenarios since this was enabled. We shouldn't take any action on it until after base is shifted to 12-STABLE. If problems persist then we may want to back this out (and #9544)
Updated by Jim Pingle over 3 years ago
- Status changed from Feedback to New
- Target version deleted (
2.5.0)
This requires RADIX_MPATH in the kernel which proved to be too unstable, thus had to be removed. See #9544.
We will revisit it in the future.
multipath option was removed in pfsense:d56f80bbbb64c83dc9bd5d05772f3773145a14f4
Updated by Jens Groh about 1 year ago
Jim Pingle wrote in #note-3:
This requires RADIX_MPATH in the kernel which proved to be too unstable, thus had to be removed. See #9544.
We will revisit it in the future.
multipath option was removed in pfsense:d56f80bbbb64c83dc9bd5d05772f3773145a14f4
Is that an option for usage in 2.7/23.01 and foward? With the new FBSD14 base would it be possible to revisit this? Multipath in FRR would be a huge gain when builing e.g. a multi-VPN router via various WAN uplinks to "bond" them together for a better throughput. Currently that works OK'ish with the loadbalanced gateway group feature but OSPF+MPATH should be better and faster to react to changes in that concern.
Updated by Jim Pingle about 1 year ago
- % Done changed from 100 to 0
Builds based on FreeBSD 14 (including 23.01 and snapshots of 2.7.0) have ROUTE_MPATH enabled in the kernel and it's active already. You can check the value of net.route.multipath. If it's 0 then it's inactive, 1 is active.
While that is in the kernel, we haven't yet enabled it in FRR for this release, but may consider it post-release in development snapshots where it can be tested with less danger.
Updated by Jim Pingle 10 months ago
- Has duplicate Feature #14464: BGP ECMP added
Updated by Mike Moore 10 months ago
Will it be enabled in any development snapshots maybe for 23.09 or made available sooner?
Updated by Mike Moore 10 months ago
i do have a use case with 2x DIA circuits. Would love to test if possible.
Updated by Alhusein Zawi 10 months ago
net.route.multipath =1 is active
[2.7.0-DEVELOPMENT][admin@pfSense.home.arpa]/root: sysctl net.route
net.route.netisr_maxqlen: 1024
net.route.debug.rtsock_debug_level: 6
net.route.debug.rt_helpers_debug_level: 6
net.route.debug.route_ctl_debug_level: 6
net.route.debug.nhop_ctl_debug_level: 6
net.route.debug.nhop_debug_level: 6
net.route.debug.nhgrp_ctl_debug_level: 6
net.route.debug.nhgrp_debug_level: 6
net.route.ipv6_nexthop: 1
net.route.hash_outbound: 0
net.route.multipath: 1
net.route.algo.debug_level: 5
.
.
.
2.7.0-DEVELOPMENT (amd64)
built on Mon Jun 05 06:04:49 UTC 2023
FreeBSD 14.0-CURRENT
Updated by Mike Moore 10 months ago
Looks like its available in the 23.05 release i am running a 6100. So its enabled but not exposed through the GUI. I have to use the cli to configure - yes?
/root: sysctl net.route
net.route.netisr_maxqlen: 1024
net.route.debug.rtsock_debug_level: 6
net.route.debug.rt_helpers_debug_level: 6
net.route.debug.route_ctl_debug_level: 6
net.route.debug.nhop_ctl_debug_level: 6
net.route.debug.nhop_debug_level: 6
net.route.debug.nhgrp_ctl_debug_level: 6
net.route.debug.nhgrp_debug_level: 6
net.route.ipv6_nexthop: 1
net.route.hash_outbound: 0
net.route.multipath: 1
net.route.algo.debug_level: 5
net.route.algo.inet.algo: radix4_lockless
net.route.algo.inet.algo_list: bsearch4, radix4_lockless, radix4
net.route.algo.inet6.algo: radix6_lockless
net.route.algo.inet6.algo_list: radix6_lockless, radix6
net.route.algo.fib_max_sync_delay_ms: 1000
net.route.algo.bucket_change_threshold_rate: 500
net.route.algo.bucket_time_ms: 50
Updated by Mike Moore 10 months ago
Confirmed that multipath is enabled by default.
Although unlikely for me, is there a way to turn OFF multipath behavior.
The workaround solution would be to use bgp attributes such as LOCAL PREF or AS-PATH PRENDING which would in effect remove ecmp but i would prefer a knob to turn.
= 10.30.1.0/24 10.6.106.10 0 31898 i
> 10.6.106.6 0 31898 i
= 10.30.2.0/24 10.6.106.10 0 31898 i
> 10.6.106.6 0 31898 i
Updated by Jim Pingle 10 months ago
Mike Moore wrote in #note-13:
Confirmed that multipath is enabled by default.
Although unlikely for me, is there a way to turn OFF multipath behavior.
The workaround solution would be to use bgp attributes such as LOCAL PREF or AS-PATH PRENDING which would in effect remove ecmp but i would prefer a knob to turn.
You should be able to disable it via tunable (net.route.multipath=0).
The muiltipath option is not enabled in the FRR package build, however, so multipath likely won't have any full effect on BGP/OSPF/etc either way yet.
Updated by Alex Kolesnik 10 months ago
Jim, are you aware of any plans to enable multipath in the FRR package?
Updated by Jim Pingle 9 months ago
- Status changed from New to Feedback
- % Done changed from 0 to 100
Turns out it's already enabled in the current builds. FRR without the "multipath" option allows 16 duplicate routes, that option ups it to 64.
Since the support is already in the kernel, it should be working now on Plus 23.05.1 and CE 2.7.0.
Updated by Chris Baker 9 months ago
Jim Pingle wrote in #note-16:
Turns out it's already enabled in the current builds. FRR without the "multipath" option allows 16 duplicate routes, that option ups it to 64.
Since the support is already in the kernel, it should be working now on Plus 23.05.1 and CE 2.7.0.
I've been trying to get multipath working in CE 2.7.0 to load balance DNS across two DNS servers but it's not working. As far as I can tell the config is correct unless I'm missing something. Troubleshooting details are in https://forum.netgate.com/topic/181816/load-balancing-with-bgp-multipath
Updated by Jim Pingle 9 months ago
- Project changed from pfSense Packages to pfSense
- Category changed from FRR to Routing
- Status changed from Feedback to Resolved
- Plus Target Version set to 23.05.1
- Release Notes set to Default
From our local testing here on Plus (23.05.1, 23.09 snaps) and CE (2.7.0, 2.8.0 snaps), with both static and BGP it appears to be working, however, be aware that the OS computes outbound flow hashing for connections. What that means is, similar to lagg, you may only see connections/packets taking the alternate paths if they are different in some way, such as different protocols, src/dst IP address combinations, and TCP/UDP connection port pairs. For example, testing with ICMP only from one to the other with no variation may never see flows take another path. The hashing takes the 5-tuple connection property set "(proto, src, dst, srcport, dstport)" into account.
If the sysctl oid for net.route.multipath is 1 and both routes show in the table, that should be enough to know it's prepared to work. You can check the nexthop data with netstat -4onW and nexthop group data with netstat -4OW and both of those should show both gateways and that they belong to the same "group".
If it does not appear to be working for you, keep the discussion on the forum and try some alternate means of testing (e.g. multiple clients and not from the firewall(s) directly).
Updated by Jim Pingle 9 months ago
- Precedes Todo #1521: Multipath Routing GUI Support added
Updated by Jim Pingle 9 months ago
- Subject changed from Enable MULTIPATH in FRR to Enable Multipath Routing in the Kernel