Bug #10465
closedpossible routing performance regression due to non use of ip_tryforward
0%
Description
A few years back Netgate sponsored upstream enhancements to FreeBSD which replaced ip_fastforward() with ip_tryforward() - which subsequently appeared in pfSense 2.3.
Whilst researching FreeBSD tuning (https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2) I noticed the recommendation to disable ICMP redirects.
This requirement comes from a patch applied to FreeBSD in August 2018 to fix ICMP redirects ... FreeBSD 11 Stable patch https://svnweb.freebsd.org/base?view=revision&revision=338343
As far as I can tell pfSense 2.4.5 has both IPv4 & IPv6 ICMP redirects defaulting to on - which based on above patch would now appear to disable tryforward path:
net.inet.ip.redirect: 1
net.inet6.ip6.redirect: 1
The workaround is obviously trivial:
sysctl net.inet.ip.redirect=0
sysctl net.inet6.ip6.redirect=0
Like many of us I'm away from my office and so lack test equipment... so I did a rudimentary analysis based on CPU
Sample vmstat 2 output from APU2 (default) - test throughput ~85Mb/s WAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 18750 319 38839 2 14 84 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 16177 276 33722 1 15 84 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 14841 139 31430 1 13 86 0 0 0 655M 3.4G 4 0 0 0 0 9 0 0 15865 109 33206 0 13 87 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 16218 130 33776 1 14 85 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 14252 109 30215 1 10 89 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 15339 106 32168 1 13 86
Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~85Mb/s WAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 18644 111 37189 0 15 84 0 0 0 655M 3.4G 2 0 0 0 0 6 0 0 16840 129 34442 1 13 87 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 17776 118 36026 1 12 87 0 0 0 655M 3.4G 4 0 0 0 0 6 0 0 16221 109 33478 0 13 86 0 0 0 655M 3.4G 6 0 0 0 0 6 0 0 17251 127 35166 1 12 87 0 0 0 655M 3.4G 2 0 0 0 0 6 0 0 18232 136 37086 1 13 86 0 0 0 655M 3.4G 5 0 0 0 0 6 0 1 18724 202 37911 1 14 85
Not entirely scientific but shows a modest 1-2% CPU drop. Clearly for GbE bandwidths the improvement is likely to be larger.
Assuming these results can be confirmed then maybe sysctl defaults can be changed in future release?
Updated by David Burns over 4 years ago
Another rudimentary analysis for single TCP connection LAN IP -> LAN VLAN IP (ie same interface routing)
Sample vmstat 2 output from APU2 (default) - test throughput ~380Mb/s LAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 669M 3.4G 436 0 0 0 281 17 0 15 2360 653 6767 4 28 68 0 0 0 669M 3.4G 438 0 0 0 281 18 0 7 2115 680 6313 4 27 69 1 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2897 1159 7995 8 27 65 0 0 0 669M 3.4G 436 0 0 0 281 18 0 0 2213 870 6490 5 27 69 1 0 0 669M 3.4G 98 0 0 0 97 26 0 0 1827 415 5718 3 14 83 0 0 0 669M 3.4G 340 0 0 0 184 17 0 0 3112 355 8214 1 7 92 1 0 0 669M 3.4G 438 0 0 2 281 17 0 1 3420 648 8646 4 22 75 0 0 0 669M 3.4G 439 0 0 0 281 9 0 0 1914 893 5858 4 22 74 0 0 0 669M 3.4G 438 0 0 0 289 17 0 2 1635 681 5290 4 22 74 1 0 0 671M 3.4G 63 0 0 0 52 18 0 0 1474 369 4959 4 23 73 0 0 0 669M 3.4G 378 0 0 0 230 18 0 0 2250 468 6497 4 25 71 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2279 728 6596 4 25 71 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1770 649 5600 5 25 71 0 0 0 669M 3.4G 436 0 0 0 281 18 0 0 1092 671 4235 4 24 71 1 0 0 669M 3.4G 4 0 0 0 0 18 0 1 1032 295 4125 4 25 71 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1558 582 5199 4 25 71 0 0 0 678M 3.4G 9606 0 0 17 9971 18 0 2 2204 10051 6944 8 33 59 0 0 0 669M 3.4G 12214 0 0 8 12172 17 0 0 2767 4081 7850 7 25 68
Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~410Mb/s LAN->LAN
procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 0 0 0 669M 3.4G 4 0 0 0 4 17 0 1 2602 169 7092 4 22 74 0 0 0 669M 3.4G 438 0 0 0 281 17 0 0 901 658 3998 4 19 76 0 0 0 669M 3.4G 436 0 0 1 285 17 0 5 880 744 3923 4 21 75 0 0 0 669M 3.4G 437 0 0 0 281 17 0 0 972 674 4056 5 20 76 1 0 0 669M 3.4G 435 0 0 0 281 17 0 0 2044 714 6066 5 23 72 0 0 0 669M 3.4G 437 0 0 0 281 17 0 0 2399 625 6869 3 13 84 0 0 0 669M 3.4G 2 0 0 0 0 17 0 0 1676 133 5373 1 5 94 1 0 0 669M 3.4G 439 0 0 0 281 17 0 0 1292 1005 4743 4 15 81 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1564 651 5218 4 20 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 1817 703 5635 4 21 75 1 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2477 652 6827 5 23 73 0 0 0 669M 3.4G 4 0 0 0 0 17 0 0 2514 183 6899 4 22 74 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 991 700 4163 4 20 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 16 1179 652 4461 4 19 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 7 929 698 3982 4 20 76 0 0 0 669M 3.4G 436 0 0 0 281 17 0 0 2111 630 6318 3 15 82
This example more clearly shows a 4-5% drop in system CPU for single TCP connection.
Expect system CPU savings to scale with number of cores for normal routing workloads.
Updated by Jim Pingle over 4 years ago
I'm not sure we should change the default to disable ICMP redirects. A modest performance gain would be nice but most users expect the behavior provided by that feature by default. Instead, we could add a GUI option to control whether ICMP redirects are enabled/disabled (default: enabled) with notes about the pros/cons of doing so, and put some info in the docs to match.
Updated by David Burns over 4 years ago
I understand your reluctance to vary FreeBSD defaults... however here is a brief summary of pfSense / FreeBSD behaviour in relation to fastforward/tryforward ICMP Redirects:
- pfSense 2.2.x (& earlier) - FreeBSD 10.1 - routing optimisation ip_fastforward (sysctl net.inet.ip.fastforwarding default off) - ICMP Redirect support default - yes
- pfSense 2.3.x - FreeBSD 10.3 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
- pfSense 2.4.0-2.4.3 - FreeBSD 11.1 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
- pfSense 2.4.4 - FreeBSD 11.2 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support default unavailable due to FreeBSD limitation
- pfSense 2.4.5 - FreeBSD 11.3-STABLE - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC & ICMP Redirects enabled) - ICMP Redirect support default - yes
So pfSense 2.3-2.4.4 users have been successfully using ip_tryforward path for quite a while with no ICMP Redirects.
In any case your suggestion to add a GUI option with explanatory text would be the best option (under System / Advanced / Networking perhaps?).
Edit: clarified ICMP Redirection support issue is in upstream FreeBSD releases 10.3-11.2
Thanks
Updated by Jim Pingle over 4 years ago
ICMP redirects have been on by default in pfSense for as long as I can remember, though there may have been a bug or two in the past which made them not work for everyone (Like #9235). We don't necessarily follow the FreeBSD default, we follow what users expect for default behavior. I looked back over 10 years in the repo and we've set the value net.inet.ip.redirect=1
and net.inet6.ip6.redirect=1
the whole time.
Users can already change the values easily on the system tunables tab, but making that even simpler with a checkbox and explanation is likely better for users.
Updated by David Burns over 4 years ago
The issue I believe has always been with FreeBSD base - and not pfSense defaults.
The MFC patch (https://svnweb.freebsd.org/base?view=revision&revision=338343) to FreeBSD 11.3-STABLE underlined that ICMP Redirects have never worked for fast path routing with ip_tryforward() in FreeBSD 10.3-11.2. In older FreeBSD releases (10.2 & earlier) with ip_fastforward() you could at least disable the fast path to enable support of ICMP Redirects.
Agreed though a return to RFC compliant defaults may be sound.
Thanks
Updated by Kevin Mychal Ong over 3 years ago
Is this issue still applicable with 2.5 or should I re-enable ICMP redirect?
Updated by David Burns over 3 years ago
- Fixed a network performance regression in the fast forwarding path with IP redirects enabled NG4965
Possibly Netgate have applied patch from https://svnweb.freebsd.org/base?view=revision&revision=367740
Hopefully they will confirm this.
Updated by Jim Pingle over 3 years ago
- Status changed from New to Resolved
- Target version set to 2.5.0
Yes, this is fixed in 21.02/2.5.0