https://redmine.pfsense.org/https://redmine.pfsense.org/favicon.ico?16780521162015-07-10T07:18:16ZpfSense bugtrackerpfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=191072015-07-10T07:18:16ZSteve Wheeler
<ul></ul><p>Seems likely to be this:<br />"Unfortunately, RSS is usually capable of hashing IPv4 and IPv4 traffic (L3+L4). All other traffic like PPPoE or MPLS or .. is usually received by queue 0." <br /><a class="external" href="https://wiki.freebsd.org/NetworkPerformanceTuning">https://wiki.freebsd.org/NetworkPerformanceTuning</a></p>
<p>There is a patch suggested there.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=203892015-09-03T21:51:53ZJim Thompsonjim@netgate.com
<ul><li><strong>Tracker</strong> changed from <i>Bug</i> to <i>Feature</i></li></ul> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=222782015-11-11T12:39:36ZJim Thompsonjim@netgate.com
<ul><li><strong>Assignee</strong> set to <i>Jim Thompson</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li><li><strong>Target version</strong> changed from <i>2.3</i> to <i>Future</i></li></ul><p>Fixing this likely requires an in-kernel RSS (Toeplitz) implementation. Such a thing is coming for FreeBSD (Adrian is working on it for the upper layers of the stack), but it's going to be a while before it's ready to interface to netisr.</p>
<p>Priority dropped to "low". Will review when we're based on 11.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=273252016-05-31T11:05:44ZJulien REVERTjulien.revert@revertconsulting.com
<ul></ul><p>Any news on this task? I'm deploying fiber internet on many pfsense APU2C4 and bandwith from wan is limited to 350 Mbits as soon as PPPoE is used (1 or the 4 cpu is 100% by PPPoE process). Otherwise (DHCP), fiber is at the top 950Mbits.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=290382016-10-10T18:59:07ZTravis Erdmann
<ul></ul><p>Now that FreeBSD 11 is out and PPPoE Gig internet is becoming more available, can we take another look at this?</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=290552016-10-11T23:19:08ZSebastian Foss
<ul></ul><p>Travis Erdmann wrote:</p>
<blockquote>
<p>Now that FreeBSD 11 is out and PPPoE Gig internet is becoming more available, can we take another look at this?</p>
</blockquote>
<p>Also happens on latests 2.4 Dev builds, but FreeBSD 11 includes the correct RSS awareness in the igb driver now.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=300612016-12-22T07:26:14ZJim Pingle
<ul><li><strong>Target version</strong> changed from <i>Future</i> to <i>2.4.0</i></li></ul><p>It still happens on 2.4, actually it's a little worse since it doesn't appear to transmit on the additional queues like it did previously:</p>
<pre>
dev.igb.1.queue1.rx_packets: 0
dev.igb.1.queue1.tx_packets: 0
dev.igb.1.queue0.rx_packets: 1978785
dev.igb.1.queue0.tx_packets: 1959503
</pre>
<p>Earlier note on the ticket said to review once we're on FreeBSD 11, so I'll set the target to 2.4 but it may need to be pushed again depending on what we find.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=301252016-12-24T19:44:44ZChris Allen
<ul></ul><p>I would like to add that I am also experiencing this issue. I would love to see this fixed in pfSense 2.4 if possible. Jim Thompson do you have any idea what might be involved to fix this now that the FreeBSD 11 driver for igb has RSS awareness?</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=303072017-01-03T11:42:38ZVladimir Suhhanovw0wkin@gmail.com
<ul></ul><p>According to this <br /><a class="external" href="https://lists.freebsd.org/pipermail/freebsd-net/2013-May/035564.html">https://lists.freebsd.org/pipermail/freebsd-net/2013-May/035564.html</a><br />Script that can solve CPU 1-core overload problem, but not default queue problem.<br /><pre>
#!/bin/sh
# PROVIDE: cpuset-igb
# REQUIRE: FILESYSTEMS
# BEFORE: netif
# KEYWORD: nojail
case "$1" in
*start)
echo "Binding igb(4) IRQs to CPUs"
cpus=`sysctl -n kern.smp.cpus`
vmstat -ai | sed -E '/^irq.*que/!d; s/^irq([0-9]+): igb([0-9]+):que ([0-9]+).*/\1 \2 \3/' |\
while read irq igb que
do
cpuset -l $(( ($igb+$que) % $cpus )) -x $irq
done
;;
esac
</pre></p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=312512017-02-09T14:15:20ZChris Allen
<ul></ul><p>Hi Jim, just wondering if this is still something that might make it into pfSense 2.4.0? I would love to use the maximum speed of my Gigabit PPPoE Fibre connection on my APU2C4 :)</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=314732017-02-11T19:57:38ZJim Thompsonjim@netgate.com
<ul></ul><p>Unlikely</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=318752017-03-01T15:08:11ZJ P
<ul></ul><p>This should definitely be marked as a bug and not a feature.<br />Has anybody tried the igb driver patch from <a class="external" href="https://wiki.freebsd.org/NetworkPerformanceTuning">https://wiki.freebsd.org/NetworkPerformanceTuning</a> ?<br />The link is dead and I can't find any archived copies. That, in combination with the script above, should at least fix this temporarily for users of NICs that use the igb driver.</p>
<p>I found a forum post (<a class="external" href="https://forum.pfsense.org/index.php?topic=114123.0">https://forum.pfsense.org/index.php?topic=114123.0</a>) that shows someone using the igb driver that is reporting all queues are used with PPPoE. Can someone test if this actually still happens?</p>
<p>Here is the FreeBSD bug tracker pointing out this issue <a class="external" href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856</a><br />The issue is over a year old (October 2015) and hasn't seen much progress.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=319462017-03-02T16:06:52ZDavid Wood
<ul></ul><p>The problem with the patch mentioned in comment 12 was that it was a kludge for igb(4) only, not a fix for the underlying issue which, so far as I remember, affects PPPoE on all multi-queue NICs. I'm not even sure the kludge will apply cleanly on FreeBSD 11.x, let alone work, as there are new in-kernel RSS features and various fixes/updates to the igb(4) driver. I suspect the patch has disappeared from a combination of failing to be a universal solution, lack of support and code rot.</p>
<p>I'm fairly certain that the pfSense team will take the approach of fixing this issue properly for all scenarios or doing nothing. An unsupported patch for a single NIC driver is unlikely to qualify for inclusion in a production firewall distribution.</p>
<p>Annoying as this issue undoubtedly is, I expect it will miss the cut for pfSense 2.4-RELEASE. There comes a point where developers have to decide an upcoming release is feature complete in order to close out the remaining bugs and ship.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=325692017-04-19T20:35:51ZChris Allen
<ul></ul><p>Could we please have this changed from "Feature" to "Bug"?</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=325702017-04-19T21:15:15ZJim Pingle
<ul></ul><p>It isn't a bug, it's a missing feature.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=328662017-05-23T09:46:05ZScott Baugher
<ul></ul><p>I'm using the nightly builds (2.4.0.b.20170522.1522 as of right now). I also use gigabit fiber over PPPoE, so I'm happy to test and report back once a fix is pushed.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=330872017-06-25T09:45:48ZJulien REVERTjulien.revert@revertconsulting.com
<ul></ul><p>Scott Baugher wrote:</p>
<blockquote>
<p>I'm using the nightly builds (2.4.0.b.20170522.1522 as of right now). I also use gigabit fiber over PPPoE, so I'm happy to test and report back once a fix is pushed.</p>
</blockquote>
<p>Is it fix using nightly builds?</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=331032017-06-27T10:25:02ZScott Baugher
<ul></ul><p>As of the June 2, 2017 build, it does not look like it. Receiving over PPPoE is still limited to one queue.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=331092017-06-28T03:06:55ZJim Thompsonjim@netgate.com
<ul><li><strong>Target version</strong> changed from <i>2.4.0</i> to <i>Future</i></li></ul> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=359752018-03-05T03:56:28ZYorick Gersie
<ul></ul><p>Took me a while to land on this issue. I'm facing similar issues not being able to utilize my full PPPoE WAN speed. Upload 500mbit/s+ Download "capped" at 350mbit/s. Any plans for addressing this issue?</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=359802018-03-05T13:06:56ZScott Baugher
<ul></ul><p>Not as of a couple of months ago. I contacted pfSense tech support (since I was using their hardware) and was basically told that they had no additional information on when it might be fixed other than what is posted here. I asked them to bring this issue up with the pfSense developers to attempt to raise its priority and get it resolved, and was basically told no. I've since experienced a failure of my pfSense hardware. That, coupled with no movement on this bug, caused me to switch away from pfSense entirely.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=360932018-03-16T12:34:40ZBenoit Lelievre
<ul></ul><p>Now that the Spectre and Meltdown patches are coming out on various OSes this becomes even more critical to fix because it's no longer possible to run a gigabit PPPoE connection off of a single processor core, no matter what speed that processor runs at.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=360942018-03-16T12:46:08ZJim Pingle
<ul></ul><p>Since this is a missing feature in FreeBSD networking, you should lobby there for it to be addressed, not here:</p>
<p><a class="external" href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856</a></p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=360972018-03-16T13:05:21ZBenoit Lelievre
<ul></ul><p>I will but I was hoping that pfSense people would also push FreeBSD on it, since I'm sure they have a much stronger and direct influence than some Internet rando.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=376132018-08-09T15:57:03ZMatthew Staver
<ul></ul><p>There has been a flurry of activity on this freebsd bug post. It sounds like the issue is a hardware limitation in the intel NIC and the proposed work around is changing net.isr.dispatch to "deferred".</p>
<p><a class="external" href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11</a></p>
<p>It looks like at some point that parameter was set in a prior release: <a class="external" href="https://redmine.pfsense.org/issues/4754">https://redmine.pfsense.org/issues/4754</a></p>
<p>As of 2.4.2 it is set to "direct".</p>
<p>Could the pfSense team weigh in on this in light of the comments from the FreeBSD guys?</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=376222018-08-10T15:15:39ZJim Pingle
<ul></ul><p>We have set it to <code>deferred</code> in the past on i386 to avoid a crash it otherwise encountered, but we do not explicitly set that tunable to any value these days.</p>
<p>Given the history there I would be cautious about changing it yourself to test. Certainly not something I would do by default for everyone without a lot of testing. If I recall correctly, changing that might lead to network performance issues of other kinds or potentially problems with ALTQ/Limiters, IPsec traffic handling, and so on.</p>
<p>I'm not sure any solid conclusion has been reached on that FreeBSD bug either, there is still some question about how it behaves on other operating systems and what the differences are there.</p>
<p>If you'd like to configure that, you can set it as a system tunable under <strong>System > Advanced</strong>, <strong>System Tunables</strong> tab. If you choose to do that, however, I would remove or disable any active ALTQ and limiter configurations.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=376392018-08-11T15:39:08ZMatthew Staver
<ul></ul><p>Jim Pingle wrote:</p>
<blockquote>
<p>If you'd like to configure that, you can set it as a system tunable under <strong>System > Advanced</strong>, <strong>System Tunables</strong> tab. If you choose to do that, however, I would remove or disable any active ALTQ and limiter configurations.</p>
</blockquote>
<p>I'm not using any traffic shaping at the moment so I went ahead and switched to deferred. With the setting direct, I get about 680Mb down and 920Mb up. Switching to deferred gained about 100Mbit of downstream (770Mbit) speed with no change in upstream.</p>
<p>If I understand the discussion over at FreeBSD, it still isn't using the other queues and that is reflected in my testing:</p>
<p>dev.igb.0.queue7.rx_packets: 0<br />dev.igb.0.queue6.rx_packets: 0<br />dev.igb.0.queue5.rx_packets: 0<br />dev.igb.0.queue4.rx_packets: 0<br />dev.igb.0.queue3.rx_packets: 0<br />dev.igb.0.queue2.rx_packets: 0<br />dev.igb.0.queue1.rx_packets: 0<br />dev.igb.0.queue0.rx_packets: 672066401</p>
<p>I'll leave the setting in place and see if any other problems present themselves. The FreeBSD guys seem pretty resistant to addressing this. Is it possible to be addressed by the pfSense team and if so how likely would that be?</p>
<p>Thanks,</p>
<p>Matt</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=376402018-08-11T15:45:50ZJim Pingle
<ul></ul><p>It is highly unlikely we'd be able to dedicate any resources toward adding this feature internally.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=380072018-08-24T00:00:05ZAlexandre Paradisalexandre.paradis@gmail.com
<ul></ul><p>ix driver seems to be affected also :</p>
<p>dev.ix.0.queue1.rx_packets: 67<br />dev.ix.0.queue1.tx_packets: 4107155<br />dev.ix.0.queue0.rx_packets: 6872678<br />dev.ix.0.queue0.tx_packets: 3345036</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=380272018-08-25T10:38:54ZSebastian Foss
<ul></ul><p>As i understood from Intel specification on various chipsets, non-ip traffic like pppoe can't be hashed for RSS to work thus always going to queue 0.</p>
<p>There are Intel 700 Series NICs that can RSS PPPoE traffic using DDP configuration file loaded from DPDK. The PPPOE pkgo profile files have been added to the Intel website <-- In case any of you need to handle 40Gbit/s PPPOE traffic...</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=382632018-09-05T21:26:08ZAlexandre Paradisalexandre.paradis@gmail.com
<ul></ul><p>You were probably kidding, but my ISP will propose exactly this (40 gig) in 2-3 years ... at least that's the plan.</p>
<p>Is DDP working in pfsense with an intel or chelsio capable (i think T4 and up?) card ? if yes, this would solve the current issue we have.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=389332018-10-09T11:43:09ZL H
<ul></ul><p>Adding the tunable: net.isr.dispatch=deferred fixed it for me to reach the full rated speed of my link.</p>
<p>Read the FreeBSD bug comment for background:<br /><a class="external" href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11</a></p>
<p>Haven't tested any side-effects of this yet, but so far it seems benign.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=392302018-11-13T18:24:10ZValentin N
<ul></ul><p>Testing net.isr.dispatch on the NetGate SG-4860 on a 1 Gbps PPPoE connection (each result is averaged across 10 runs):<br /><pre>
Test Direct#1 Direct#2 Deferred#1 Deferred#2
Avg (Mbps) 530.98 494.19 620.68 708.50
Max (Mbps) 582.13 565.41 759.45 790.44
</pre></p>
<p>In my case the improvement was not quite line rate, but it is noticeable.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=392702018-11-15T15:26:23ZJim Pingle
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>Added info to the docs about using the sysctl tunable to work around this. There doesn't appear to be anything more we can do.</p>
<p><a class="external" href="https://www.netgate.com/docs/pfsense/hardware/tuning-and-troubleshooting-network-cards.html#pppoe-with-multi-queue-nics">https://www.netgate.com/docs/pfsense/hardware/tuning-and-troubleshooting-network-cards.html#pppoe-with-multi-queue-nics</a></p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=420822019-08-21T13:49:33ZJim Pingle
<ul><li><strong>Target version</strong> deleted (<del><i>Future</i></del>)</li></ul> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=428002019-11-04T18:21:58ZChris Collins
<ul></ul><p>Interestingly I appear to have rss working on pppoe using igb driver.</p>
<p>the tx is very misbalanced about 10:1 but rx is almost 50:50.</p> pfSense - Feature #4821: PPPoE WANs do not take full advantage of NIC driver queues for receiving traffichttps://redmine.pfsense.org/issues/4821?journal_id=503122021-01-13T15:51:47Z→ luckman212luke.hamburg@gmail.com
<ul></ul><p>Can we get some kind of CAPTCHA on here to rid ourselves of this polluting junk??</p>