Project

General

Profile

Bug #10465

possible routing performance regression due to non use of ip_tryforward

Added by David Burns 12 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Routing
Target version:
2.5.0
Start date:
04/16/2020
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.5
Affected Architecture:
Release Notes:
Default

Description

A few years back Netgate sponsored upstream enhancements to FreeBSD which replaced ip_fastforward() with ip_tryforward() - which subsequently appeared in pfSense 2.3.

Whilst researching FreeBSD tuning (https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2) I noticed the recommendation to disable ICMP redirects.
This requirement comes from a patch applied to FreeBSD in August 2018 to fix ICMP redirects ... FreeBSD 11 Stable patch https://svnweb.freebsd.org/base?view=revision&revision=338343

As far as I can tell pfSense 2.4.5 has both IPv4 & IPv6 ICMP redirects defaulting to on - which based on above patch would now appear to disable tryforward path:
net.inet.ip.redirect: 1
net.inet6.ip6.redirect: 1

The workaround is obviously trivial:
sysctl net.inet.ip.redirect=0
sysctl net.inet6.ip6.redirect=0

Like many of us I'm away from my office and so lack test equipment... so I did a rudimentary analysis based on CPU
Sample vmstat 2 output from APU2 (default) - test throughput ~85Mb/s WAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 18750   319 38839  2 14 84
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 16177   276 33722  1 15 84
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 14841   139 31430  1 13 86
0 0 0 655M  3.4G     4   0   0   0     0    9   0   0 15865   109 33206  0 13 87
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 16218   130 33776  1 14 85
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 14252   109 30215  1 10 89
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 15339   106 32168  1 13 86

Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~85Mb/s WAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 18644   111 37189  0 15 84
0 0 0 655M  3.4G     2   0   0   0     0    6   0   0 16840   129 34442  1 13 87
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 17776   118 36026  1 12 87
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 16221   109 33478  0 13 86
0 0 0 655M  3.4G     6   0   0   0     0    6   0   0 17251   127 35166  1 12 87
0 0 0 655M  3.4G     2   0   0   0     0    6   0   0 18232   136 37086  1 13 86
0 0 0 655M  3.4G     5   0   0   0     0    6   0   1 18724   202 37911  1 14 85

Not entirely scientific but shows a modest 1-2% CPU drop. Clearly for GbE bandwidths the improvement is likely to be larger.

Assuming these results can be confirmed then maybe sysctl defaults can be changed in future release?

History

#1 Updated by David Burns 12 months ago

Another rudimentary analysis for single TCP connection LAN IP -> LAN VLAN IP (ie same interface routing)

Sample vmstat 2 output from APU2 (default) - test throughput ~380Mb/s LAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 669M  3.4G   436   0   0   0   281   17   0  15 2360   653  6767  4 28 68
0 0 0 669M  3.4G   438   0   0   0   281   18   0   7 2115   680  6313  4 27 69
1 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2897  1159  7995  8 27 65
0 0 0 669M  3.4G   436   0   0   0   281   18   0   0 2213   870  6490  5 27 69
1 0 0 669M  3.4G    98   0   0   0    97   26   0   0 1827   415  5718  3 14 83
0 0 0 669M  3.4G   340   0   0   0   184   17   0   0 3112   355  8214  1  7 92
1 0 0 669M  3.4G   438   0   0   2   281   17   0   1 3420   648  8646  4 22 75
0 0 0 669M  3.4G   439   0   0   0   281    9   0   0 1914   893  5858  4 22 74
0 0 0 669M  3.4G   438   0   0   0   289   17   0   2 1635   681  5290  4 22 74
1 0 0 671M  3.4G    63   0   0   0    52   18   0   0 1474   369  4959  4 23 73
0 0 0 669M  3.4G   378   0   0   0   230   18   0   0 2250   468  6497  4 25 71
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2279   728  6596  4 25 71
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1770   649  5600  5 25 71
0 0 0 669M  3.4G   436   0   0   0   281   18   0   0 1092   671  4235  4 24 71
1 0 0 669M  3.4G     4   0   0   0     0   18   0   1 1032   295  4125  4 25 71
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1558   582  5199  4 25 71
0 0 0 678M  3.4G  9606   0   0  17  9971   18   0   2 2204 10051  6944  8 33 59
0 0 0 669M  3.4G 12214   0   0   8 12172   17   0   0 2767  4081  7850  7 25 68

Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~410Mb/s LAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 669M  3.4G     4   0   0   0     4   17   0   1 2602   169  7092  4 22 74
0 0 0 669M  3.4G   438   0   0   0   281   17   0   0  901   658  3998  4 19 76
0 0 0 669M  3.4G   436   0   0   1   285   17   0   5  880   744  3923  4 21 75
0 0 0 669M  3.4G   437   0   0   0   281   17   0   0  972   674  4056  5 20 76
1 0 0 669M  3.4G   435   0   0   0   281   17   0   0 2044   714  6066  5 23 72
0 0 0 669M  3.4G   437   0   0   0   281   17   0   0 2399   625  6869  3 13 84
0 0 0 669M  3.4G     2   0   0   0     0   17   0   0 1676   133  5373  1  5 94
1 0 0 669M  3.4G   439   0   0   0   281   17   0   0 1292  1005  4743  4 15 81
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1564   651  5218  4 20 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1817   703  5635  4 21 75
1 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2477   652  6827  5 23 73
0 0 0 669M  3.4G     4   0   0   0     0   17   0   0 2514   183  6899  4 22 74
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0  991   700  4163  4 20 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0  16 1179   652  4461  4 19 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0   7  929   698  3982  4 20 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2111   630  6318  3 15 82

This example more clearly shows a 4-5% drop in system CPU for single TCP connection.

Expect system CPU savings to scale with number of cores for normal routing workloads.

#2 Updated by Jim Pingle 12 months ago

I'm not sure we should change the default to disable ICMP redirects. A modest performance gain would be nice but most users expect the behavior provided by that feature by default. Instead, we could add a GUI option to control whether ICMP redirects are enabled/disabled (default: enabled) with notes about the pros/cons of doing so, and put some info in the docs to match.

#3 Updated by David Burns 12 months ago

I understand your reluctance to vary FreeBSD defaults... however here is a brief summary of pfSense / FreeBSD behaviour in relation to fastforward/tryforward ICMP Redirects:

  • pfSense 2.2.x (& earlier) - FreeBSD 10.1 - routing optimisation ip_fastforward (sysctl net.inet.ip.fastforwarding default off) - ICMP Redirect support default - yes
  • pfSense 2.3.x - FreeBSD 10.3 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
  • pfSense 2.4.0-2.4.3 - FreeBSD 11.1 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
  • pfSense 2.4.4 - FreeBSD 11.2 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support default unavailable due to FreeBSD limitation
  • pfSense 2.4.5 - FreeBSD 11.3-STABLE - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC & ICMP Redirects enabled) - ICMP Redirect support default - yes

So pfSense 2.3-2.4.4 users have been successfully using ip_tryforward path for quite a while with no ICMP Redirects.

In any case your suggestion to add a GUI option with explanatory text would be the best option (under System / Advanced / Networking perhaps?).

Edit: clarified ICMP Redirection support issue is in upstream FreeBSD releases 10.3-11.2

Thanks

#4 Updated by Jim Pingle 12 months ago

ICMP redirects have been on by default in pfSense for as long as I can remember, though there may have been a bug or two in the past which made them not work for everyone (Like #9235). We don't necessarily follow the FreeBSD default, we follow what users expect for default behavior. I looked back over 10 years in the repo and we've set the value net.inet.ip.redirect=1 and net.inet6.ip6.redirect=1 the whole time.

Users can already change the values easily on the system tunables tab, but making that even simpler with a checkbox and explanation is likely better for users.

#5 Updated by David Burns 12 months ago

The issue I believe has always been with FreeBSD base - and not pfSense defaults.

The MFC patch (https://svnweb.freebsd.org/base?view=revision&revision=338343) to FreeBSD 11.3-STABLE underlined that ICMP Redirects have never worked for fast path routing with ip_tryforward() in FreeBSD 10.3-11.2. In older FreeBSD releases (10.2 & earlier) with ip_fastforward() you could at least disable the fast path to enable support of ICMP Redirects.

Agreed though a return to RFC compliant defaults may be sound.

Thanks

#6 Updated by Kevin Mychal Ong about 2 months ago

Is this issue still applicable with 2.5 or should I re-enable ICMP redirect?

#7 Updated by David Burns about 2 months ago

The 21.02 / 2.5 Release Notes lists this fix (in the Operating System section):
  • Fixed a network performance regression in the fast forwarding path with IP redirects enabled NG4965

Possibly Netgate have applied patch from https://svnweb.freebsd.org/base?view=revision&revision=367740

Hopefully they will confirm this.

#8 Updated by Jim Pingle about 2 months ago

  • Status changed from New to Resolved
  • Target version set to 2.5.0

Yes, this is fixed in 21.02/2.5.0

Also available in: Atom PDF