Project

General

Profile

Bug #3666

PMTUD is broken for NATed traffic

Added by Chris Buechler about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Ermal Luçi
Category:
Operating System
Target version:
Start date:
05/17/2014
Due date:
% Done:

0%

Affected Version:
2.2
Affected Architecture:

Description

Where you have an interface on a system with a lower MTU than other interfaces, send traffic larger than the MTU of the egress interface, and have pf enabled, you end up with a PMTUD black hole.

For example, simple LAN and WAN setup:
- set LAN to 1500 MTU and WAN to 1000
- send something from a host on LAN destined to something on WAN that's bigger than WAN's MTU with DF set, such as with hping:
hping3 -y -d 1400 -2 -p 12345 $dest_IP

where 12345 is the port and $dest_IP is an IP or hostname to send the traffic.

With pf enabled, the packet just disappears, the client gets nothing back. Disable pf, and you get back the appropriate "frag needed, DF set" ICMP error.

The above description and symptoms are specific to 2.2/10-STABLE, as of the most recent snapshot available at the time of this writing.

broken-rules.debug (6.43 KB) Chris Buechler, 06/11/2014 05:59 AM

broken-pfctl-vvsr.txt Magnifier (19.2 KB) Chris Buechler, 06/11/2014 05:59 AM

broken-pfctl-vvsn.txt Magnifier (2.09 KB) Chris Buechler, 06/11/2014 05:59 AM

Associated revisions

Revision 415b71f1
Added by Ermal Luçi over 3 years ago

Fixes #3666. Set the sysctl net.inet.icmp.reply_from_interface to 1 to use the incoming interface to send the icmp reply from. It uses another part of patch to pf to undo NAT if it was already performed before

Revision c46f9695
Added by Ermal Luçi over 3 years ago

Actually make default sysctls reside on globals.inc and use those by default this allows to trim down the config.xml sysctl and also fixes #3666 by setting set source interface on reply of icmp

History

#1 Updated by Ermal Luçi almost 4 years ago

  • Status changed from New to Feedback

Patch put in to try to handle this case.

For record this is happening due to NAT being applied on packets and the generated ICMP is targeted to the pfSense machine itself.

#2 Updated by Chris Buechler almost 4 years ago

  • Subject changed from enabling pf breaks PMTUD to PMTUD is broken for NATed traffic
  • Status changed from Feedback to New

no change. I did confirm it's specific to NATed traffic and updated subject accordingly. Send any packet > egress interface's MTU with pf enabled where it's NATing the traffic, and you get no reply (and it doesn't leave any other interface on the box either). Turn off NAT, or disable pf, and you get the appropriate dest unreachable, frag needed reply.

#3 Updated by Chris Buechler almost 4 years ago

Additional data point. This seemingly isn't an issue in stock FreeBSD 10-STABLE. One I had handy:

FreeBSD m6600-vbox-10stable 10.0-STABLE FreeBSD 10.0-STABLE #0 r265018: Sun Apr 27 19:01:41 UTC 2014     root@grind.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Same scenario with NAT, that system works fine. Seems a regression related to changes we make.

#4 Updated by Ermal Luçi almost 4 years ago

You used the same ruleset on stock FreeBSD as pfSense?

#5 Updated by Chris Buechler almost 4 years ago

not identical, no. Had the same basic components - scrub all, pass all, nat on. I can throw the completely identical ruleset on there if that'd help. Can't reach that system at this instant to try it.

#6 Updated by Chris Buechler almost 4 years ago

I think you're on to something there. This:

scrub all 

nat on em0 from 192.168.0.0/16 to any -> (em0)

pass in all 
pass out all 

seems to work fine.

The attached, a pretty stock allow all ruleset for what we generate, doesn't work. I haven't tried to narrow it down any further, suggestions on likely culprits?

#7 Updated by Jim Thompson almost 4 years ago

  • Assignee set to Ermal Luçi

#8 Updated by Ermal Luçi over 3 years ago

  • Status changed from New to Feedback

I think the sysctl that was activated should fix this.

#9 Updated by Ermal Luçi over 3 years ago

  • % Done changed from 0 to 100

#10 Updated by Chris Buechler over 3 years ago

  • Status changed from Feedback to New

no change in behavior

#11 Updated by Ermal Luçi over 3 years ago

Actually try out a next coming snapshot i found a quick hack that will help even for icmp case.

#12 Updated by Ermal Luçi over 3 years ago

  • Status changed from New to Feedback

Did you try to send an tcp/udp packet rather than icmp one?

#13 Updated by Chris Buechler over 3 years ago

  • Status changed from Feedback to Confirmed

no change. Ermal, msg me and we can both take a look at my test setup.

#14 Updated by Chris Buechler over 3 years ago

Ermal - no change with the kernel you built. I have a test setup up now that you can reach. /msg me for info.

#15 Updated by Ermal Luçi over 3 years ago

  • Status changed from Confirmed to Feedback

#16 Updated by Ermal Luçi over 3 years ago

Teh reply from interface was not being set properly.
Works for me now.

#17 Updated by Chris Buechler over 3 years ago

  • Status changed from Feedback to Confirmed
  • % Done changed from 100 to 0

no change. Test setup on dev ESX is fully in place now, info on chaos wiki.

#18 Updated by Chris Buechler over 3 years ago

  • Status changed from Confirmed to Resolved

scratch that, the test box wasn't rebooted post-gitsync and gitsync doesn't apply the relevant change on the fly. This works now

Also available in: Atom PDF