Bug #9643
closed
Limiters do not function properly on 2.5 snapshots
Added by Greg M almost 5 years ago. Updated about 3 years ago.
0%
Description
Hi all!
Discussion here: https://forum.netgate.com/topic/145091/quick-question-about-limiters
I think there is a bug in limiters...
Problem #1:
When I create limiters and set floating firewall rules for WAN interface, all traffic from all LANs stops working. If I create floating rules
with same limiters for LAN interfaces, everything works.
I was following forum discussion samples and tutorials, YouTube videos etc... Limiters were set up properly and floating rules as well.
Problem #2:
With limiter on LAN ifaces all traffic on LAN iface is limited to 80 Mbit/s which is OK.
But when I start to download torrent for example, LAN stays at 80 Mbit/s but WAN goes above 80 and eventually reaches 90 Mbit/s.
There is no other sources of traffic just torrents. One WAN and 2 LAN.
See image: https://forum.netgate.com/assets/uploads/files/1563691216842-capture.png
Because of problem #2, I was trying to limit WAN iface instead of LAN to avoid this issue but when I did that, it blocked all traffic completely.
Pfsense is on Hyper-V.
Using latest snapshot as of today.
We can continue on forums to avoid spamming here, but I think something is not quite right with limiters on 2.5.0 snaps.
Thanks!
Files
Updated by Greg M almost 5 years ago
Hi again.
I restored config on 2.4.4-p3 and this are working just fine there.
I believe this on is related to https://redmine.pfsense.org/issues/8954#change-41002 .
Thanks!
Updated by Jim Pingle almost 5 years ago
- Category set to Traffic Shaper (Limiters)
- Target version set to 2.5.0
The two cannot be related. ALTQ is not used for limiters.
I have also seen a similar situation on 2.5 where limiters were not functional and had to be removed to pass traffic.
Updated by Greg M almost 5 years ago
Hmmm OK, I have Hyper-V, 2.5.0 and pppoe.
But weird is, that on when applied on IN direction on LAN it works ok.
Updated by Jim Pingle over 4 years ago
- Subject changed from Limiters in 2.5 to Limiters do not function properly on 2.5 snapshots
Updated by Grant Peier over 4 years ago
I experienced the same behavior as Greg M when updating from 2.4.4-p3 to 2.5.0. This was on a bare-metal install.
Updated by Jim Pingle over 4 years ago
Nothing yet, but since we are rebasing on FreeBSD 12.1 soon, it will need to wait until after that happens.
Updated by Ryl Thelandria about 4 years ago
Experiencing same behavior as reported by Greg M on my physical install of pfsense 2.5 dev. Traffic just stops.
Followed the instructions from 2.4.4 here (from a pfsense Short Topic vid, first topic): https://www.youtube.com/watch?v=o8nL81DzTlU&t=380 No dice.
Meantime, I disabled the floating rule for now, and I try again every few weeks...no change in behavior since I first tried back at the start of Jan 2020.
Any news? Thanks, in advance. Love your software otherwise! Gorgeous, powerful, versatile!
Updated by Gyula Kelemen almost 4 years ago
Hi.
Same behavior on Proxmox/KVM - pfSense 2.5.0.a.20200518.1031 with vtnet driver.
Any update on this?
Thanks!
Updated by Manuel Piovan almost 4 years ago
not working for me either
2.5.0.a.20200522.0732
I need to disable the floating rule to make internet work again
Updated by Tom Fuke almost 4 years ago
I'm having the same issue, running on a VK-T40E:
2.5.0.a.20200603.1253
If I enable the floating rule, I lose all internet/WAN traffic.
Out of interest, for those running on 2.4.x, can you see the queue in any of the pfSense monitoring/status/log tools?
Updated by Chris Collins over 3 years ago
Hi guys, just to confirm it looks like I have the same problem.
pfSense running in a Proxmox VM, I did gui update from 2.4.x to 2.5 devel.
After an hour I was informed server's behind firewall are dead, and noticed dummynet was making all traffic just timeout as if there was a block rule in place, the limiter was configured as in the pfSense video guide.
I am using vtnet driver.
Updated by Chris Collins over 3 years ago
I can also confirm it works fine on LAN, and since the setup uses NAT, it means I can use this as a workaround, I put myself forward as a volunteer if any patches get made and need testing, I can test on this server.
Updated by Luiz Souza over 3 years ago
- Status changed from New to Feedback
Can someone confirm this is still broken with a current snapshot ?
I was able to set up a floating rule and the limits were properly applied.
Updated by Abhinav Tella over 3 years ago
Still broken for me on the latest build, I tested just now.
Updated by Luiz Souza over 3 years ago
Can you give me more details ? show me your rules and results ?
Updated by Abhinav Tella over 3 years ago
Here are the limiters and firewall floating rule I used. When the firewall rule is enabled, no traffic gets through the WAN, the same setup worked fine in 2.4.5-P1. I have no other user generated rulesets other than default from install time. No pfBlockerng or any other packages. I am testing bare metal on an AMD Epyc 3251 SuperMicro build w/Intel X710-T2L ethernet adapter.
UPDATE: The upload limiter seems to work, I went back to the firewall rule and selected none for the "Out Pipe". So basically it's the downlink limiter that's not functional in 2.5.0.
Limiters:
Download Limiter:
Bandwidth: 1000 Mbps (I get 1150-1200 Mbps from ISP) (later tried 900 Mbps)
Queue Management Algorithm: CoDel
Scheduler: FQ_Codel
Queue Length: 1000 also tried leaving blank
ECN: Enabled
—Download Queue:
Queue Management Algorithm: CoDel
ECN: Enabled
Upload Limiter:
Bandwidth: 36 Mbps
Queue Management Algorithm: CoDel
Scheduler: FQ_Codel
Queue Length: 1000 also tried leaving blank
ECN: Enabled
—Upload Queue:
Queue Management Algorithm: CoDel
ECN: Enabled
Firewall Floating Rule:
Action: Pass
Interface: WAN
Direction: Out
Address Family: IPv4
Protocol: Any
Advanced:
Gateway: WAN_DHCP - 192.168.x.x
In/Out Pipe: Upload Queue (In) / Download Queue (Out)
Updated by Renato Botelho over 3 years ago
- Status changed from Feedback to New
- Assignee set to Luiz Souza
Updated by Jesse Beauclaire over 3 years ago
Abhinav Tella wrote:
Here are the limiters and firewall floating rule I used. When the firewall rule is enabled, no traffic gets through the WAN, the same setup worked fine in 2.4.5-P1. I have no other user generated rulesets other than default from install time. No pfBlockerng or any other packages. I am testing bare metal on an AMD Epyc 3251 SuperMicro build w/Intel X710-T2L ethernet adapter.
UPDATE: The upload limiter seems to work, I went back to the firewall rule and selected none for the "Out Pipe". So basically it's the downlink limiter that's not functional in 2.5.0.
Limiters:
Download Limiter:
Bandwidth: 1000 Mbps (I get 1150-1200 Mbps from ISP) (later tried 900 Mbps)
Queue Management Algorithm: CoDel
Scheduler: FQ_Codel
Queue Length: 1000 also tried leaving blank
ECN: Enabled
—Download Queue:
Queue Management Algorithm: CoDel
ECN: EnabledUpload Limiter:
Bandwidth: 36 Mbps
Queue Management Algorithm: CoDel
Scheduler: FQ_Codel
Queue Length: 1000 also tried leaving blank
ECN: Enabled
—Upload Queue:
Queue Management Algorithm: CoDel
ECN: EnabledFirewall Floating Rule:
Action: Pass
Interface: WAN
Direction: Out
Address Family: IPv4
Protocol: Any
Advanced:
Gateway: WAN_DHCP - 192.168.x.x
In/Out Pipe: Upload Queue (In) / Download Queue (Out)
I am also having the same issue running the identical configuration as Abhinav Tella on 2.5.0-DEVELOPMENT (amd64) (built on Thu Sep 10 01:03:22 EDT 2020). The only difference is that I have it split between two floating rules; one for IPv4 and the other for IPv6.
Updated by Thomas Pilgaard over 3 years ago
Tested fq-codel out on the latest snapshot and found out that if i apply an outbound WAN pass rule to ipv6 it does apply and works just fine for IPv6 traffic. But at the same time IPv4 traffic gets instant blocked even though there isn't a rule that apply for it
Updated by Samuel Hanna over 3 years ago
- File Screenshot_82.png Screenshot_82.png added
I've tested FQ_CODEL Too, but not working.
i have dual wan setup, and i have 4 different limiters (2) for every wan connection for UP and Down.
when i make a Floating rule with WANs interfaces by OUT direction it works fine by just adding the IN QUEUE (which is the upload limiter queue) and set the OUT QUEUE for none.
but if i added the OUT QUEUE for a value (which is the download limiter queue) everything is not working, so i tried to mask destination hosts to know the line of data, which really surprised me that (in the download limiter queue the destinations were lan IP addresses not the interface IP) which i think is playing role in blocking data through the Queue.
i can really make a video of it. wish i could help.
Updated by Anonymous about 3 years ago
- Affected Version changed from 2.5.0 to All
Updated by Anonymous about 3 years ago
- Affected Version changed from All to 2.5.0
Updated by Peter Grehan about 3 years ago
- Private changed from No to Yes
I'm able to reproduce this. As mentioned in earlier comments, the issue only shows when the inbound queue is enabled. Will be doing some more tracing of ipfw/dummynet to see where packets are being dropped.
Updated by Kevin S about 3 years ago
Hi. I am also able to reproduce this. It works fine on 2.4.5, but on 2.5.0, the minute the floating rule is enable, I lose access.
Peter Grehan wrote:
I'm able to reproduce this. As mentioned in earlier comments, the issue only shows when the inbound queue is enabled. Will be doing some more tracing of ipfw/dummynet to see where packets are being dropped.
Updated by Luiz Souza about 3 years ago
- Status changed from In Progress to Feedback
All the fixes from 2.4.5 are now merged.
Initial tests looks good.
Updated by Sish Kitane about 3 years ago
Luiz Souza wrote:
All the fixes from 2.4.5 are now merged.
Initial tests looks good.
I can confirm this is resolved for me on 2.5.0.r.20210210.0933.
Thank you and congrats on the release!
Updated by Renato Botelho about 3 years ago
- Status changed from Feedback to Resolved
Updated by Jim Pingle about 3 years ago
Looks good here as well. Not only can I pass traffic with limiters on, I am back to an A on the bufferbloat test thanks to CoDel.
Updated by Jesse Beauclaire about 3 years ago
I'm not sure if this is related, my understanding of this is limited. After creating/enabling CODEL traffic limiters again after updating pfSense to 2.5.0 RC and applying floating rules for IPv4 and IPv6, I receive the following error message:
There were error(s) loading the rules: /tmp/rules.debug:326: no routing address with matching address family found. - The line in question reads [326]: pass out quick on { igb0 } $GWWAN_DHCP6 inet6 from any to any tracker 1609523960 keep state dnqueue( 2,1) label "USER_RULE: CoDel Limiters"
@ 2021-02-11 07:39:56
If I disable the IPv6 floating rule for the limiter I do not receive the error message, but upon re-enabling the rule the error comes back.
Updated by Jim Pingle about 3 years ago
That doesn't appear to be related to this specific issue, it looks like a problem with your rule / state of your system.
Updated by Jesse Beauclaire about 3 years ago
Jim Pingle wrote:
That doesn't appear to be related to this specific issue, it looks like a problem with your rule / state of your system.
Any guidance on where I could look for assistance so I don't clog up this bug report?
Updated by Jim Pingle about 3 years ago
Make a post on the forum and discuss it there, that's the best way to diagnose your issue.
Updated by Chris Collins about 3 years ago
I updated to the latest stable (new RC 2.5)
Sadly I still have the same problem, I am still checking stuff to make sure I haven't broken it from previous fiddling to try and get it working when it initially broke but early signs are not good.
Its basically floating match rule on WAN interface, quick match, assign the limiter, and all traffic hitting the rule is going into a black hole.
Updated by Chris Collins about 3 years ago
Ok to summarise.
It works on outbound WAN matching (this was broken before the patch).
It works on inbound and outbound LAN matching.
It goes into a black hole on inbound WAN matching.
If I keep the match rule but remove the dummynet queue (so it just matches but does nothing elsE), the traffic is processed normally, so it is a dummynet issue on inbound WAN traffic.
Updated by Jesse Beauclaire about 3 years ago
Chris Collins wrote:
Ok to summarise.
It works on outbound WAN matching (this was broken before the patch).
It works on inbound and outbound LAN matching.
It goes into a black hole on inbound WAN matching.If I keep the match rule but remove the dummynet queue (so it just matches but does nothing elsE), the traffic is processed normally, so it is a dummynet issue on inbound WAN traffic.
Is that with both IPv4 and IPv6 rules?
Updated by Chris Collins about 3 years ago
I only have it configured with ipv4.
Updated by Jim Pingle about 3 years ago
Chris Collins wrote:
It goes into a black hole on inbound WAN matching.
If I keep the match rule but remove the dummynet queue (so it just matches but does nothing elsE), the traffic is processed normally, so it is a dummynet issue on inbound WAN traffic.
Was that same configuration known to work on 2.4.5-p1? That doesn't sound like something we've supported with limiters (there are some quirks with match+dummynet behavior) -- Nobody else is reporting issues, including people with limiters on multiple WANs, so it may be you have stumbled onto a case that shouldn't work which wouldn't be a valid test for this bug.
Updated by Chris Collins about 3 years ago
It did work yes, the reason for the configuration is, the firewall is in front of a webserver, and I want people who download the files to not have an advantage if they use something like a download manager to multithread, so I shape per ip, and I only want it to shape on webserver traffic, so I match the connections inbound.
If you want, I can probably setup another pfsense VM fairly quickly on 2.4.5 just to confirm this for this issue.
I am able to do the shaping on LAN though so it isnt the end of the world if its not working, but you may possibly get other reports if anyone else uses it the same way as I was.
Updated by Jim Pingle about 3 years ago
At this point I'd say open a new and more specific bug report for that once you have all the info collected and re-tested, since this issue was for limiters being completely broken for everybody and that is a much more niche/focused case which seems unrelated. Something else in the base OS changing might have affected that differently.
Updated by Chris Collins about 3 years ago
Yeah I can do that at a later date. I will keep it out of this report now.
Updated by Ashus CZ about 3 years ago
I believe I have the same issue, I just upgraded from 2.4.5 to 2.5.0 and upload queues are empty.
I also use multi-WAN configuration.
An example of floating rule that matched HTTPS traffic:
Action: match
Interface: WAN1
Direction: any
Family: IPv4+IPv6
Protocol: TCP
Source: any
Destination: any, port 443
Ackqueue: qACK1
Queue: qNormal1
Another one was identical with these differences:
Interface: WAN2
Ackqueue: qACK2
Queue: qNormal2
Queues have this structure:
Interface WAN1 -qUploadV1 --qACK1 --qICMP1 --qRealtime1 --qHigh1 --qNormal1 --qLow1 Interface WAN2 -qUploadV2 --qACK2 --qICMP2 --qRealtime2 --qHigh2 --qNormal2 --qLow2 Interface LAN -qLink --qDownloadV1 ---qACK1 ---qICMP1 ---qRealtime1 ---qHigh1 ---qNormal1 ---qLow1 --qDownloadV2 ---qACK2 ---qICMP2 ---qRealtime2 ---qHigh2 ---qNormal2 ---qLow2
On version 2.4.5 and before, download and upload queues were filled with this traffic. On 2.5.0 the upload queues are empty, effectively not shaping upload at all causing increased latency over DSL lines.
Any advice please?