Bug #10465
possible routing performance regression due to non use of ip_tryforward
0%
Description
A few years back Netgate sponsored upstream enhancements to FreeBSD which replaced ip_fastforward() with ip_tryforward() - which subsequently appeared in pfSense 2.3.
Whilst researching FreeBSD tuning (https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2) I noticed the recommendation to disable ICMP redirects.
This requirement comes from a patch applied to FreeBSD in August 2018 to fix ICMP redirects ... FreeBSD 11 Stable patch https://svnweb.freebsd.org/base?view=revision&revision=338343
As far as I can tell pfSense 2.4.5 has both IPv4 & IPv6 ICMP redirects defaulting to on - which based on above patch would now appear to disable tryforward path:
net.inet.ip.redirect: 1
net.inet6.ip6.redirect: 1
The workaround is obviously trivial:
sysctl net.inet.ip.redirect=0
sysctl net.inet6.ip6.redirect=0
Like many of us I'm away from my office and so lack test equipment... so I did a rudimentary analysis based on CPU
Sample vmstat 2 output from APU2 (default) - test throughput ~85Mb/s WAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 18750 319 38839 2 14 84 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 16177 276 33722 1 15 84 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 14841 139 31430 1 13 86 0 0 0 655M 3.4G 4 0 0 0 0 9 0 0 15865 109 33206 0 13 87 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 16218 130 33776 1 14 85 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 14252 109 30215 1 10 89 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 15339 106 32168 1 13 86
Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~85Mb/s WAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 18644 111 37189 0 15 84 0 0 0 655M 3.4G 2 0 0 0 0 6 0 0 16840 129 34442 1 13 87 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 17776 118 36026 1 12 87 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 16221 109 33478 0 13 86 0 0 0 655M 3.4G 6 0 0 0 0 6 0 0 17251 127 35166 1 12 87 0 0 0 655M 3.4G 2 0 0 0 0 6 0 0 18232 136 37086 1 13 86 0 0 0 655M 3.4G 5 0 0 0 0 6 0 1 18724 202 37911 1 14 85
Not entirely scientific but shows a modest 1-2% CPU drop. Clearly for GbE bandwidths the improvement is likely to be larger.
Assuming these results can be confirmed then maybe sysctl defaults can be changed in future release?
History
#1
Updated by David Burns 3 months ago
Updated by Another rudimentary analysis for single TCP connection LAN IP -> LAN VLAN IP (ie same interface routing)
Sample vmstat 2 output from APU2 (default) - test throughput ~380Mb/s LAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 669M 3.4G 436 0 0 0 281 17 0 15 2360 653 6767 4 28 68 0 0 0 669M 3.4G 438 0 0 0 281 18 0 7 2115 680 6313 4 27 69 1 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2897 1159 7995 8 27 65 0 0 0 669M 3.4G 436 0 0 0 281 18 0 0 2213 870 6490 5 27 69 1 0 0 669M 3.4G 98 0 0 0 97 26 0 0 1827 415 5718 3 14 83 0 0 0 669M 3.4G 340 0 0 0 184 17 0 0 3112 355 8214 1 7 92 1 0 0 669M 3.4G 438 0 0 2 281 17 0 1 3420 648 8646 4 22 75 0 0 0 669M 3.4G 439 0 0 0 281 9 0 0 1914 893 5858 4 22 74 0 0 0 669M 3.4G 438 0 0 0 289 17 0 2 1635 681 5290 4 22 74 1 0 0 671M 3.4G 63 0 0 0 52 18 0 0 1474 369 4959 4 23 73 0 0 0 669M 3.4G 378 0 0 0 230 18 0 0 2250 468 6497 4 25 71 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2279 728 6596 4 25 71 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1770 649 5600 5 25 71 0 0 0 669M 3.4G 436 0 0 0 281 18 0 0 1092 671 4235 4 24 71 1 0 0 669M 3.4G 4 0 0 0 0 18 0 1 1032 295 4125 4 25 71 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1558 582 5199 4 25 71 0 0 0 678M 3.4G 9606 0 0 17 9971 18 0 2 2204 10051 6944 8 33 59 0 0 0 669M 3.4G 12214 0 0 8 12172 17 0 0 2767 4081 7850 7 25 68
Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~410Mb/s LAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 669M 3.4G 4 0 0 0 4 17 0 1 2602 169 7092 4 22 74 0 0 0 669M 3.4G 438 0 0 0 281 17 0 0 901 658 3998 4 19 76 0 0 0 669M 3.4G 436 0 0 1 285 17 0 5 880 744 3923 4 21 75 0 0 0 669M 3.4G 437 0 0 0 281 17 0 0 972 674 4056 5 20 76 1 0 0 669M 3.4G 435 0 0 0 281 17 0 0 2044 714 6066 5 23 72 0 0 0 669M 3.4G 437 0 0 0 281 17 0 0 2399 625 6869 3 13 84 0 0 0 669M 3.4G 2 0 0 0 0 17 0 0 1676 133 5373 1 5 94 1 0 0 669M 3.4G 439 0 0 0 281 17 0 0 1292 1005 4743 4 15 81 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1564 651 5218 4 20 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1817 703 5635 4 21 75 1 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2477 652 6827 5 23 73 0 0 0 669M 3.4G 4 0 0 0 0 17 0 0 2514 183 6899 4 22 74 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 991 700 4163 4 20 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 16 1179 652 4461 4 19 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 7 929 698 3982 4 20 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2111 630 6318 3 15 82
This example more clearly shows a 4-5% drop in system CPU for single TCP connection.
Expect system CPU savings to scale with number of cores for normal routing workloads.
#2
Updated by Jim Pingle 3 months ago
Updated by I'm not sure we should change the default to disable ICMP redirects. A modest performance gain would be nice but most users expect the behavior provided by that feature by default. Instead, we could add a GUI option to control whether ICMP redirects are enabled/disabled (default: enabled) with notes about the pros/cons of doing so, and put some info in the docs to match.
#3
Updated by David Burns 3 months ago
I understand your reluctance to vary FreeBSD defaults... however here is a brief summary of pfSense / FreeBSD behaviour in relation to fastforward/tryforward ICMP Redirects:
- pfSense 2.2.x (& earlier) - FreeBSD 10.1 - routing optimisation ip_fastforward (sysctl net.inet.ip.fastforwarding default off) - ICMP Redirect support default - yes
- pfSense 2.3.x - FreeBSD 10.3 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
- pfSense 2.4.0-2.4.3 - FreeBSD 11.1 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
- pfSense 2.4.4 - FreeBSD 11.2 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support default unavailable due to FreeBSD limitation
- pfSense 2.4.5 - FreeBSD 11.3-STABLE - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC & ICMP Redirects enabled) - ICMP Redirect support default - yes
So pfSense 2.3-2.4.4 users have been successfully using ip_tryforward path for quite a while with no ICMP Redirects.
In any case your suggestion to add a GUI option with explanatory text would be the best option (under System / Advanced / Networking perhaps?).
Edit: clarified ICMP Redirection support issue is in upstream FreeBSD releases 10.3-11.2
Thanks
#4
Updated by Jim Pingle 3 months ago
ICMP redirects have been on by default in pfSense for as long as I can remember, though there may have been a bug or two in the past which made them not work for everyone (Like #9235). We don't necessarily follow the FreeBSD default, we follow what users expect for default behavior. I looked back over 10 years in the repo and we've set the value net.inet.ip.redirect=1 and net.inet6.ip6.redirect=1 the whole time.
Users can already change the values easily on the system tunables tab, but making that even simpler with a checkbox and explanation is likely better for users.
#5
Updated by David Burns 3 months ago
The issue I believe has always been with FreeBSD base - and not pfSense defaults.
The MFC patch (https://svnweb.freebsd.org/base?view=revision&revision=338343) to FreeBSD 11.3-STABLE underlined that ICMP Redirects have never worked for fast path routing with ip_tryforward() in FreeBSD 10.3-11.2. In older FreeBSD releases (10.2 & earlier) with ip_fastforward() you could at least disable the fast path to enable support of ICMP Redirects.
Agreed though a return to RFC compliant defaults may be sound.
Thanks