Bug #10465: possible routing performance regression due to non use of ip_tryforward - pfSense - pfSense bugtracker

Actions

Copy link

Bug #10465

closed

possible routing performance regression due to non use of ip_tryforward

Added by David Burns about 2 years ago. Updated about 1 year ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Routing

Target version:

2.5.0

Start date:

04/16/2020

Due date:

% Done:

Estimated time:

Plus Target Version:

Release Notes:

Affected Version:

2.4.5

Affected Architecture:

Description

A few years back Netgate sponsored upstream enhancements to FreeBSD which replaced ip_fastforward() with ip_tryforward() - which subsequently appeared in pfSense 2.3.

Whilst researching FreeBSD tuning (https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2) I noticed the recommendation to disable ICMP redirects.
This requirement comes from a patch applied to FreeBSD in August 2018 to fix ICMP redirects ... FreeBSD 11 Stable patch https://svnweb.freebsd.org/base?view=revision&revision=338343

As far as I can tell pfSense 2.4.5 has both IPv4 & IPv6 ICMP redirects defaulting to on - which based on above patch would now appear to disable tryforward path:
net.inet.ip.redirect: 1
net.inet6.ip6.redirect: 1

The workaround is obviously trivial:
sysctl net.inet.ip.redirect=0
sysctl net.inet6.ip6.redirect=0

Like many of us I'm away from my office and so lack test equipment... so I did a rudimentary analysis based on CPU
Sample vmstat 2 output from APU2 (default) - test throughput ~85Mb/s WAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 18750   319 38839  2 14 84
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 16177   276 33722  1 15 84
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 14841   139 31430  1 13 86
0 0 0 655M  3.4G     4   0   0   0     0    9   0   0 15865   109 33206  0 13 87
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 16218   130 33776  1 14 85
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 14252   109 30215  1 10 89
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 15339   106 32168  1 13 86

Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~85Mb/s WAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 18644   111 37189  0 15 84
0 0 0 655M  3.4G     2   0   0   0     0    6   0   0 16840   129 34442  1 13 87
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 17776   118 36026  1 12 87
0 0 0 655M  3.4G     4   0   0   0     0    6   0   0 16221   109 33478  0 13 86
0 0 0 655M  3.4G     6   0   0   0     0    6   0   0 17251   127 35166  1 12 87
0 0 0 655M  3.4G     2   0   0   0     0    6   0   0 18232   136 37086  1 13 86
0 0 0 655M  3.4G     5   0   0   0     0    6   0   1 18724   202 37911  1 14 85

Not entirely scientific but shows a modest 1-2% CPU drop. Clearly for GbE bandwidths the improvement is likely to be larger.

Assuming these results can be confirmed then maybe sysctl defaults can be changed in future release?

Actions

Copy link

Updated by David Burns about 2 years ago

Another rudimentary analysis for single TCP connection LAN IP -> LAN VLAN IP (ie same interface routing)

Sample vmstat 2 output from APU2 (default) - test throughput ~380Mb/s LAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 669M  3.4G   436   0   0   0   281   17   0  15 2360   653  6767  4 28 68
0 0 0 669M  3.4G   438   0   0   0   281   18   0   7 2115   680  6313  4 27 69
1 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2897  1159  7995  8 27 65
0 0 0 669M  3.4G   436   0   0   0   281   18   0   0 2213   870  6490  5 27 69
1 0 0 669M  3.4G    98   0   0   0    97   26   0   0 1827   415  5718  3 14 83
0 0 0 669M  3.4G   340   0   0   0   184   17   0   0 3112   355  8214  1  7 92
1 0 0 669M  3.4G   438   0   0   2   281   17   0   1 3420   648  8646  4 22 75
0 0 0 669M  3.4G   439   0   0   0   281    9   0   0 1914   893  5858  4 22 74
0 0 0 669M  3.4G   438   0   0   0   289   17   0   2 1635   681  5290  4 22 74
1 0 0 671M  3.4G    63   0   0   0    52   18   0   0 1474   369  4959  4 23 73
0 0 0 669M  3.4G   378   0   0   0   230   18   0   0 2250   468  6497  4 25 71
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2279   728  6596  4 25 71
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1770   649  5600  5 25 71
0 0 0 669M  3.4G   436   0   0   0   281   18   0   0 1092   671  4235  4 24 71
1 0 0 669M  3.4G     4   0   0   0     0   18   0   1 1032   295  4125  4 25 71
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1558   582  5199  4 25 71
0 0 0 678M  3.4G  9606   0   0  17  9971   18   0   2 2204 10051  6944  8 33 59
0 0 0 669M  3.4G 12214   0   0   8 12172   17   0   0 2767  4081  7850  7 25 68

Sample vmstat 2 output from APU2 (ICMP redirects disabled) - test throughput ~410Mb/s LAN->LAN

procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 669M  3.4G     4   0   0   0     4   17   0   1 2602   169  7092  4 22 74
0 0 0 669M  3.4G   438   0   0   0   281   17   0   0  901   658  3998  4 19 76
0 0 0 669M  3.4G   436   0   0   1   285   17   0   5  880   744  3923  4 21 75
0 0 0 669M  3.4G   437   0   0   0   281   17   0   0  972   674  4056  5 20 76
1 0 0 669M  3.4G   435   0   0   0   281   17   0   0 2044   714  6066  5 23 72
0 0 0 669M  3.4G   437   0   0   0   281   17   0   0 2399   625  6869  3 13 84
0 0 0 669M  3.4G     2   0   0   0     0   17   0   0 1676   133  5373  1  5 94
1 0 0 669M  3.4G   439   0   0   0   281   17   0   0 1292  1005  4743  4 15 81
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1564   651  5218  4 20 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 1817   703  5635  4 21 75
1 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2477   652  6827  5 23 73
0 0 0 669M  3.4G     4   0   0   0     0   17   0   0 2514   183  6899  4 22 74
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0  991   700  4163  4 20 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0  16 1179   652  4461  4 19 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0   7  929   698  3982  4 20 76
0 0 0 669M  3.4G   436   0   0   0   281   17   0   0 2111   630  6318  3 15 82

This example more clearly shows a 4-5% drop in system CPU for single TCP connection.

Expect system CPU savings to scale with number of cores for normal routing workloads.

Actions

Copy link

Updated by Jim Pingle about 2 years ago

I'm not sure we should change the default to disable ICMP redirects. A modest performance gain would be nice but most users expect the behavior provided by that feature by default. Instead, we could add a GUI option to control whether ICMP redirects are enabled/disabled (default: enabled) with notes about the pros/cons of doing so, and put some info in the docs to match.

Actions

Copy link

Updated by David Burns about 2 years ago

I understand your reluctance to vary FreeBSD defaults... however here is a brief summary of pfSense / FreeBSD behaviour in relation to fastforward/tryforward ICMP Redirects:

pfSense 2.2.x (& earlier) - FreeBSD 10.1 - routing optimisation ip_fastforward (sysctl net.inet.ip.fastforwarding default off) - ICMP Redirect support default - yes
pfSense 2.3.x - FreeBSD 10.3 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
pfSense 2.4.0-2.4.3 - FreeBSD 11.1 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support unavailable due to FreeBSD limitation
pfSense 2.4.4 - FreeBSD 11.2 - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC) - ICMP Redirect support default unavailable due to FreeBSD limitation
pfSense 2.4.5 - FreeBSD 11.3-STABLE - routing optimisation ip_tryforward (implicitly enabled - except for IPSEC & ICMP Redirects enabled) - ICMP Redirect support default - yes

So pfSense 2.3-2.4.4 users have been successfully using ip_tryforward path for quite a while with no ICMP Redirects.

In any case your suggestion to add a GUI option with explanatory text would be the best option (under System / Advanced / Networking perhaps?).

Edit: clarified ICMP Redirection support issue is in upstream FreeBSD releases 10.3-11.2

Thanks

Actions

Copy link

Updated by Jim Pingle about 2 years ago

ICMP redirects have been on by default in pfSense for as long as I can remember, though there may have been a bug or two in the past which made them not work for everyone (Like #9235). We don't necessarily follow the FreeBSD default, we follow what users expect for default behavior. I looked back over 10 years in the repo and we've set the value net.inet.ip.redirect=1 and net.inet6.ip6.redirect=1 the whole time.

Users can already change the values easily on the system tunables tab, but making that even simpler with a checkbox and explanation is likely better for users.

Actions

Copy link

Updated by David Burns about 2 years ago

The issue I believe has always been with FreeBSD base - and not pfSense defaults.

The MFC patch (https://svnweb.freebsd.org/base?view=revision&revision=338343) to FreeBSD 11.3-STABLE underlined that ICMP Redirects have never worked for fast path routing with ip_tryforward() in FreeBSD 10.3-11.2. In older FreeBSD releases (10.2 & earlier) with ip_fastforward() you could at least disable the fast path to enable support of ICMP Redirects.

Agreed though a return to RFC compliant defaults may be sound.

Thanks

Actions

Copy link