Project

General

Profile

Actions

New Content #10311

open

Default net.link.ifqmaxlen value leads to packet loss under load in OpenVPN

Added by Alexey Ab about 4 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
OpenVPN
Target version:
-
Start date:
03/02/2020
Due date:
% Done:

0%

Estimated time:

Description

Default value for net.link.ifqmaxlen is 128.

When OpenVPN is under load and transfer speed reaches 20-30 MBit, we have following errors count increasing:

VPN1_BRIDGE Interface:
In/out errors: 0/4623

netstat -idb -I bridge0 -> OERRS=4623
netstat -idb -I ovpnc6 -> DROP=4623

(Drops are important errors, but are not shown on status_interfaces.php page, please consider showing them too)

Increasing net.link.ifqmaxlen to 2048 eliminates these errors, and gives significant OpenVPN throughput increase (5x).

Also it was observed that packet drops may depend on system power management, setting CPU to high performance mode also eliminates
packet drops and gives significant OpenVPN performance increase.

Actions #1

Updated by Anonymous about 4 years ago

Commenting to watch. (Personally I expect that 128 has been chosen as good trade off between performance and keeping bufferbloat in check, but I have no actual clue.)

Actions #2

Updated by Alexey Ab about 4 years ago

I am continuing to investigate OpenVPN Performance using PFSense as virtual machine under VMWare.

Following tweaks allows OpenVPN to saturate 100 Mbit internet link with 150ms ping without packet drops.
FTP File transfer over VPN now gives stable 10-12 MBytes/s.

1) Give at least 4 CPU cores to PFSense virtual machine
2) Allow power management in the host (Windows of Linux) decrease frequency only to 75% of base frequency (in my case minimum is 2.6 Ghz).
3) Increase OpenVPN send/receive buffers to 2 MBytes
4) /boot/loader.conf.local settings:
hw.em.rx_process_limit=-1
hw.em.txd=2048
hw.em.rxd=2048
net.link.ifqmaxlen="2048"

Actions #3

Updated by Alexey Ab almost 4 years ago

Tried simple setup of PFSense 2.4.5 (without bridges, just TUN adapter) on VPS server.

Same effect - 20-30 mbit OpenVPN speed with default net.link.ifqmaxlen, and >100 Mbit with net.link.ifqmaxlen="2048".

Actions #4

Updated by Viktor Gurov over 2 years ago

This is a really useful tweak, we have to add it to https://docs.netgate.com/pfsense/en/latest/hardware/tune.html

Actions #5

Updated by Chris Collins over 1 year ago

Interestingly its forced to a value of 128 now.

If set it inside on /boot/loader.conf.local, it will apply on the next reboot but auto removed from file so the reboot after that its lost. I ended up patching /etc/inc/pfsense-utils.inc to allow the customisation, but couldnt find why this restriction was imposed as I assume there must be a reason for it.

Actions #6

Updated by Marcos M over 1 year ago

  • Tracker changed from Bug to New Content
  • Project changed from pfSense to pfSense Docs
  • Subject changed from Too low net.link.ifqmaxlen causes packet drop under load when using OpenVPN inside bridge interface under load to Default net.link.ifqmaxlen value leads to packet loss under load in OpenVPN
  • Category changed from OpenVPN to OpenVPN
  • Affected Version deleted (2.4.4-p3)
  • Affected Architecture deleted (amd64)

Chris Collins wrote in #note-5:

Interestingly its forced to a value of 128 now.

If set it inside on /boot/loader.conf.local, it will apply on the next reboot but auto removed from file so the reboot after that its lost. I ended up patching /etc/inc/pfsense-utils.inc to allow the customisation, but couldnt find why this restriction was imposed as I assume there must be a reason for it.

I was not able to reproduce this on the latest dev snapshot.

Actions #7

Updated by Chris Collins over 1 year ago

Might be quicker if I show you the code, then you check if the code that restricts it is there, I m guessing you unable to use a release version?

/etc/inc/pfsense-utils.inc

line 1264, this just sets a changed default in /boot/loader.conf

/* Set maximum send queue length. */
$data[] = 'net.link.ifqmaxlen="128"';

and line 1133, this hard codes the removal of the tunable if its configured in /boot/loader.conf.local

@ /* These values should be removed from loader.conf and loader.conf.local * As they will be replaced when necessary. */
$remove = array("hw.usb.no_pf", "hint.mdio.0.at", "hint.e6000sw.0",
"hw.e6000sw.default_disabled", "vm.pmap.pti",
"net.pf.request_maxcount", "hw.hn.vf_transparent",
"hw.hn.use_if_start", "net.link.ifqmaxlen");@

Actions #8

Updated by Chris Collins over 1 year ago

I have verified myself now on github master branch.

Looks like the code has been changed to no longer filter in loader.conf.local, only in loader.conf instead so someone recognised the problem. :)

Its been quite a while since the last stable release so a lot of this stuff I guess is taking its time to propagate down.

Actions #9

Updated by Marcos M over 1 year ago

Generally there's a decent chance that an issue has already been reported and/or resolved, hence it's good to search for issues and test snapshots to avoid duplicate work. The issue you're running into has indeed already been resolved, see https://redmine.pfsense.org/issues/12862

Actions

Also available in: Atom PDF