Project

General

Profile

Actions

Bug #9263

closed

Incorrect ICMP reply when using limiters

Added by Kirill Khazan about 5 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Category:
Traffic Shaper (Limiters)
Target version:
Start date:
01/08/2019
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
22.05
Release Notes:
Affected Version:
2.4.4
Affected Architecture:

Description

My setup is as follows. pfSense 2.4.4_p2, it maintains a L2TP tunnel to my ISP and all the traffic is configured to go through that interface (WAN_L2TP).

I configured a floating rule (this is the only rule configured, everything else is default):
Action=Pass, Quick=checked, Interface=WAN_L2TP, Direction=out, Address Family=IPV4,Protocol=Any, Gateway=WAN_L2TP_L2TP, In/Out pipes - WanUpQ, WanDownQ. The limiters are simple Tail Drop/FIFO ones, but I also tried FQ_CODEL limiters, the result is the same. With this setup, everything works as far as I can tell, other than... traceroute and mtr. It looks like this:

$ traceroute -n yahoo.com
traceroute to yahoo.com (72.30.35.10), 30 hops max, 60 byte packets
1 192.168.1.3 0.544 ms 0.510 ms 0.485 ms
2 72.30.35.10 9.542 ms 9.715 ms 9.854 ms
3 * * *
4 72.30.35.10 9.645 ms 9.693 ms 9.663 ms
5 72.30.35.10 66.242 ms 69.697 ms 66.402 ms
6 72.30.35.10 67.251 ms 67.036 ms 67.506 ms
7 72.30.35.10 75.363 ms 72.511 ms 86.762 ms
8 72.30.35.10 76.549 ms 76.740 ms 153.401 ms
9 72.30.35.10 153.355 ms 161.405 ms 161.197 ms
10 72.30.35.10 169.345 ms 158.462 ms 158.295 ms
11 72.30.35.10 166.803 ms 164.302 ms 163.989 ms
12 72.30.35.10 171.817 ms 176.850 ms 172.274 ms
13 72.30.35.10 165.496 ms 166.961 ms 181.393 ms
14 72.30.35.10 172.873 ms 170.853 ms 172.007 ms
$ mtr google.com
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.1.3 0.0% 4 0.2 0.3 0.2 0.4 0.0
2. 72.30.35.10 0.0% 4 9.0 10.8 9.0 13.3 1.6

As you can see, traceroute shows the same ip (which is the target host's ip) for every hop, but the timings are different. mtr is broken even worse, it only shows my router and then (I guess) the second hop labeled with the target's ip. Further investigating the issue by issuing pings with specific TTL values:
$ ping -t 1 google.com
PING google.com (172.217.21.238) 56(84) bytes of data.
From 192.168.1.3 (192.168.1.3) icmp_seq=1 Time to live exceeded

$ ping -t 2 google.com
PING google.com (74.125.140.101) 56(84) bytes of data.
From wq-in-f101.1e100.net (74.125.140.101) icmp_seq=1 Time to live exceeded
  1. etc, until hop 9

If I set out the "out" pipe in the rule to "none" and leave only the "in" pipe filled with WanUpQ, the problem disappears. If I configure the limiter on LAN or WAN interface, the problem disappears.

Actions #2

Updated by Jim Pingle over 4 years ago

  • Category set to Traffic Shaper (ALTQ)
Actions #3

Updated by Jim Pingle over 4 years ago

  • Category changed from Traffic Shaper (ALTQ) to Traffic Shaper (Limiters)
Actions #4

Updated by Kacper Michajłow almost 4 years ago

People on forum are wrongly attributing this issue as the same as #932 which is something quite different, omitting router in traceroute results (i.e not decreasing TTL) is something different than we see here.

As we can see below, trace result is almost fine, except that each hop have the same host assigned.

With Limiter:


  1    <1 ms    <1 ms    <1 ms  <...> [192.168.0.1]
  2     *        *        *     Request timed out.
  3    11 ms    14 ms     8 ms  one.one.one.one [1.1.1.1]
  4    12 ms    11 ms    12 ms  one.one.one.one [1.1.1.1]
  5    12 ms    14 ms    18 ms  one.one.one.one [1.1.1.1]
  6    13 ms    13 ms    11 ms  one.one.one.one [1.1.1.1]
  7    13 ms    10 ms    11 ms  one.one.one.one [1.1.1.1]
  8    55 ms    54 ms    54 ms  one.one.one.one [1.1.1.1]
  9    49 ms    49 ms    51 ms  one.one.one.one [1.1.1.1]
 10    37 ms    37 ms    56 ms  one.one.one.one [1.1.1.1]
 11    55 ms    52 ms    50 ms  one.one.one.one [1.1.1.1]

Without Limiter:


  1    <1 ms    <1 ms    <1 ms  <...> [192.168.0.1]
  2     *        *        *     Request timed out.
  <removed some hops, but they are properly shown>
  8    55 ms    53 ms    50 ms  ae-12.r24.amstnl02.nl.bb.gin.ntt.net [129.250.3.81]
  9    59 ms    54 ms    53 ms  ae-7.r03.amstnl02.nl.bb.gin.ntt.net [129.250.2.103]
 10    58 ms    59 ms    60 ms  81.20.65.150
 11    56 ms    64 ms    56 ms  one.one.one.one [1.1.1.1]

Actions #5

Updated by Miroslav Shubernetskiy almost 4 years ago

The ticket is explicitly for 2.4.4. Given that 2.4.5 is out now, the same issue is also impacting 2.4.5.

In my case I have simple CoDel limiters for both up and down on WAN and whenever limiter is enabled, traceroute is exhibiting the same behavior where all hops show the target IP. For context for my report in the forums https://forum.netgate.com/topic/152252/pfsense-rewrites-source-ip-for-icmp-errors-breaking-traceroute

Actions #6

Updated by Niccolò Marchi over 2 years ago

Same on 2.5 and 2.6

Actions #7

Updated by Marcos M almost 2 years ago

On 22.05, this seems to only happen when applying limiters on the WAN interface rather than the LAN interfaces. For example, create a floating match in rule on LAN interfaces, src/dst any, and the the issue goes away.

Actions #8

Updated by Tomasz K. almost 2 years ago

Marcos Mendoza wrote in #note-7:

On 22.05, this seems to only happen when applying limiters on the WAN interface rather than the LAN interfaces. For example, create a floating match in rule on LAN interfaces, src/dst any, and the the issue goes away.

I have the same observation. But with src/dst as any you will limit speed between your LAN interfaces. I would suggest creating RFC1918 alias and change src to RFC1918 and dst to !RFC1918

Actions #9

Updated by Marcos M almost 2 years ago

  • Status changed from New to Feedback
  • Target version set to 22.05
  • Plus Target Version set to 22.05

Tested on 22.05.a.20220510.1205 with either pass quick or match rules on either LAN or WAN interfaces. This is now fixed.

Actions #10

Updated by → luckman212 almost 2 years ago

Is there any way us mere mortals can access these snaps? Or are they still private only?

Actions #11

Updated by Marcos M almost 2 years ago

→ luckman212 wrote in #note-10:

Is there any way us mere mortals can access these snaps? Or are they still private only?

Currently private only, but stay tuned ;)

Actions #12

Updated by Jim Pingle almost 2 years ago

  • Target version changed from 22.05 to 2.7.0
Actions #13

Updated by Jim Pingle almost 2 years ago

  • Status changed from Feedback to Resolved
  • Assignee set to Kristof Provost

Assigning to Kristof since it was likely fixed along the way when moving dummynet and such info PF

Actions

Also available in: Atom PDF