Bug #9263
closed
Incorrect ICMP reply when using limiters
0%
Description
My setup is as follows. pfSense 2.4.4_p2, it maintains a L2TP tunnel to my ISP and all the traffic is configured to go through that interface (WAN_L2TP).
I configured a floating rule (this is the only rule configured, everything else is default):
Action=Pass, Quick=checked, Interface=WAN_L2TP, Direction=out, Address Family=IPV4,Protocol=Any, Gateway=WAN_L2TP_L2TP, In/Out pipes - WanUpQ, WanDownQ. The limiters are simple Tail Drop/FIFO ones, but I also tried FQ_CODEL limiters, the result is the same. With this setup, everything works as far as I can tell, other than... traceroute and mtr. It looks like this:
$ traceroute -n yahoo.com
traceroute to yahoo.com (72.30.35.10), 30 hops max, 60 byte packets
1 192.168.1.3 0.544 ms 0.510 ms 0.485 ms
2 72.30.35.10 9.542 ms 9.715 ms 9.854 ms
3 * * *
4 72.30.35.10 9.645 ms 9.693 ms 9.663 ms
5 72.30.35.10 66.242 ms 69.697 ms 66.402 ms
6 72.30.35.10 67.251 ms 67.036 ms 67.506 ms
7 72.30.35.10 75.363 ms 72.511 ms 86.762 ms
8 72.30.35.10 76.549 ms 76.740 ms 153.401 ms
9 72.30.35.10 153.355 ms 161.405 ms 161.197 ms
10 72.30.35.10 169.345 ms 158.462 ms 158.295 ms
11 72.30.35.10 166.803 ms 164.302 ms 163.989 ms
12 72.30.35.10 171.817 ms 176.850 ms 172.274 ms
13 72.30.35.10 165.496 ms 166.961 ms 181.393 ms
14 72.30.35.10 172.873 ms 170.853 ms 172.007 ms$ mtr google.com
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.1.3 0.0% 4 0.2 0.3 0.2 0.4 0.0
2. 72.30.35.10 0.0% 4 9.0 10.8 9.0 13.3 1.6As you can see, traceroute shows the same ip (which is the target host's ip) for every hop, but the timings are different. mtr is broken even worse, it only shows my router and then (I guess) the second hop labeled with the target's ip. Further investigating the issue by issuing pings with specific TTL values:
$ ping -t 1 google.com
PING google.com (172.217.21.238) 56(84) bytes of data.
From 192.168.1.3 (192.168.1.3) icmp_seq=1 Time to live exceeded
$ ping -t 2 google.com
PING google.com (74.125.140.101) 56(84) bytes of data.
From wq-in-f101.1e100.net (74.125.140.101) icmp_seq=1 Time to live exceeded- etc, until hop 9
If I set out the "out" pipe in the rule to "none" and leave only the "in" pipe filled with WanUpQ, the problem disappears. If I configure the limiter on LAN or WAN interface, the problem disappears.
Updated by Kirill Khazan almost 6 years ago
Updated by Jim Pingle over 5 years ago
- Category changed from Traffic Shaper (ALTQ) to Traffic Shaper (Limiters)
Updated by Kacper Michajłow over 4 years ago
People on forum are wrongly attributing this issue as the same as #932 which is something quite different, omitting router in traceroute results (i.e not decreasing TTL) is something different than we see here.
As we can see below, trace result is almost fine, except that each hop have the same host assigned.
With Limiter:
1 <1 ms <1 ms <1 ms <...> [192.168.0.1]
2 * * * Request timed out.
3 11 ms 14 ms 8 ms one.one.one.one [1.1.1.1]
4 12 ms 11 ms 12 ms one.one.one.one [1.1.1.1]
5 12 ms 14 ms 18 ms one.one.one.one [1.1.1.1]
6 13 ms 13 ms 11 ms one.one.one.one [1.1.1.1]
7 13 ms 10 ms 11 ms one.one.one.one [1.1.1.1]
8 55 ms 54 ms 54 ms one.one.one.one [1.1.1.1]
9 49 ms 49 ms 51 ms one.one.one.one [1.1.1.1]
10 37 ms 37 ms 56 ms one.one.one.one [1.1.1.1]
11 55 ms 52 ms 50 ms one.one.one.one [1.1.1.1]
Without Limiter:
1 <1 ms <1 ms <1 ms <...> [192.168.0.1]
2 * * * Request timed out.
<removed some hops, but they are properly shown>
8 55 ms 53 ms 50 ms ae-12.r24.amstnl02.nl.bb.gin.ntt.net [129.250.3.81]
9 59 ms 54 ms 53 ms ae-7.r03.amstnl02.nl.bb.gin.ntt.net [129.250.2.103]
10 58 ms 59 ms 60 ms 81.20.65.150
11 56 ms 64 ms 56 ms one.one.one.one [1.1.1.1]
Updated by Miroslav Shubernetskiy over 4 years ago
The ticket is explicitly for 2.4.4. Given that 2.4.5 is out now, the same issue is also impacting 2.4.5.
In my case I have simple CoDel limiters for both up and down on WAN and whenever limiter is enabled, traceroute is exhibiting the same behavior where all hops show the target IP. For context for my report in the forums https://forum.netgate.com/topic/152252/pfsense-rewrites-source-ip-for-icmp-errors-breaking-traceroute
Updated by
Updated by Marcos M over 2 years ago
On 22.05, this seems to only happen when applying limiters on the WAN interface rather than the LAN interfaces. For example, create a floating match in rule on LAN interfaces, src/dst any, and the the issue goes away.
Updated by Tomasz K. over 2 years ago
Marcos Mendoza wrote in #note-7:
On 22.05, this seems to only happen when applying limiters on the WAN interface rather than the LAN interfaces. For example, create a floating match in rule on LAN interfaces, src/dst any, and the the issue goes away.
I have the same observation. But with src/dst as any you will limit speed between your LAN interfaces. I would suggest creating RFC1918 alias and change src to RFC1918 and dst to !RFC1918
Updated by Marcos M over 2 years ago
- Status changed from New to Feedback
- Target version set to 22.05
- Plus Target Version set to 22.05
Tested on 22.05.a.20220510.1205 with either pass quick or match rules on either LAN or WAN interfaces. This is now fixed.
Updated by → luckman212 over 2 years ago
Is there any way us mere mortals can access these snaps? Or are they still private only?
Updated by Marcos M over 2 years ago
→ luckman212 wrote in #note-10:
Is there any way us mere mortals can access these snaps? Or are they still private only?
Currently private only, but stay tuned ;)
Updated by Jim Pingle over 2 years ago
- Target version changed from 22.05 to 2.7.0
Updated by Jim Pingle over 2 years ago
- Status changed from Feedback to Resolved
- Assignee set to Kristof Provost
Assigning to Kristof since it was likely fixed along the way when moving dummynet and such info PF