Project

General

Profile

Bug #8165

Fragmented at source IPv6 packets (UDP + ICMP Ping) are not forwarded / v2.4.2 AMD64

Added by Mike Nichols almost 3 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Routing
Target version:
-
Start date:
12/05/2017
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.2
Affected Architecture:
amd64

Description

This issue came to light when I encountered a problem with a SIP phone not receiving SIP Invite messages resulting in missed calls. This had previously worked OK and while I have not regression tested to find when it appeared I think it may be with v2.4.0 onward. Furthermore the problem can not be reproduced using pfSense v2.3.5 i386. I am running pfSense v2.4.2 using FreeBSD 11.1-RELEASE-p4 on a Mini-ITX based system with Celeron N2930 processor.

This problem is believed to only affect IPv6.

Packet capture shows that a SIP Invite packet arrives at the WAN interface fragmented. pfSense responds with an ICMPV6 "packet too big" message. The fragmented packets are dropped. This behaviour is not exhibited with v2.3.5 i386; the packet is correctly forwarded to LAN and no ICMPV6 "packet too big" is generated.

11:56:03.998498 IP6 2001:ab7::5 > 2a00:23c5:d007:8700:7e2f:80ff:fe20:ce8a: frag (0|1440) 5060 > 5060: UDP, bad length 1475 > 1432
11:56:03.998623 IP6 2001:ab7::5 > 2a00:23c5:d007:8700:7e2f:80ff:fe20:ce8a: frag (1440|43)
11:56:03.998735 IP6 2a00:23c5:d007:8700:230:18ff:fec9:ce4a > 2001:ab7::5: ICMP6, packet too big, mtu 1500, length 124

J Knott has independently verified the bug as follows:

FWIW, I just tried an experiment. First off, I tried pinging, with both IPv4 & IPv6, with oversize packets to a computer on the local LAN. Both worked fine. I then connected one computer to another interface on my firewall and tried again. This time IPv4 worked, but IPv6 didn't. I can see the fragmented IPv6 pings leaving one computer, but not arriving on the other. So, it appears pfSense/FreeBSD is not passing packets that have been fragmented by the source, but does pass IPv4.

packetcapture (3).cap.zip (1.38 KB) packetcapture (3).cap.zip Packet capture from WAN interface Mike Nichols, 12/05/2017 11:36 AM

History

#1 Updated by Mike Nichols almost 3 years ago

Just realised the packet capture example was truncated by one character. Here's what it should look like:

16:56:59.061983 IP6 2001:ab7::5 > 2a00:23c5:d007:f800:7e2f:80ff:fe20:ce8a: frag (0|1448) 5060 > 5060: UDP, bad length 1475 > 1440
16:56:59.062024 IP6 2001:ab7::5 > 2a00:23c5:d007:f800:7e2f:80ff:fe20:ce8a: frag (1448|35)
16:56:59.062132 IP6 2a00:23c5:d007:f800:230:18ff:fec9:ce4a > 2001:ab7::5: ICMP6, packet too big, mtu 1500, length 1240

I have adjusted the PPPoE MTU to be 1500 since the previous packet capture. It's made no difference to the bug and the fragmented UDP packets are still being erroneously dropped.

#2 Updated by Kevin A McGrail almost 3 years ago

Just wanted to file a METOO on this where it seems to be causing issues with IPv6 and UDP especially where we saw it with DNS and the DCC.

For example, we did some more troubleshooting on this. It looks like the firewall just simply isn’t allowing packet fragmentation at all. I sent a huge string over udp on that port and got a response. First we tried just a regular small string:

[2.2.6-RELEASE][]/root: echo -n “hi” | nc -6u dcc1.pccc.com 6277

Got a response on the firewall:

16:19:32.425288 e8:9a:8f:be:1b:9e > 00:15:5d:14:10:11, ethertype IPv6 (0x86dd), length 64: (flowlabel 0xf0bd1, hlim 48, next-header UDP (17) payload length: 10) 2600:1700:6020:2e80:6a05:caff:fe20:f59a.42821 > 2604:9100:7:9::1:33.6277: [udp sum ok] UDP, length 2

Good, previous problem of no UDP making it through is gone. But it looks small…let’s try a bigger string and try to stretch that stream across a huge packet:

(Forgive length here)

[2.2.6-RELEASE][]/root: echo -n "012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789" | nc -6u dcc1.pccc.com 6277

Came back with another response on the firewall, this time length 960. Still single packet we tried again, basically did the above string but 10x as long, should have seen 2 packets. No response at all, no breaking up into multiple fragments. either.

When we got to that point, that's when we found this bug report. We think we are seeing the same and the above information might help recreate the issue more readily.

#3 Updated by Kevin A McGrail almost 3 years ago

Mike Nichols wrote:

This issue came to light when I encountered a problem with a SIP phone not receiving SIP Invite messages resulting in missed calls. This had previously worked OK and while I have not regression tested to find when it appeared I think it may be with v2.4.0 onward. Furthermore the problem can not be reproduced using pfSense v2.3.5 i386. I am running pfSense v2.4.2 using FreeBSD 11.1-RELEASE-p4 on a Mini-ITX based system with Celeron N2930 processor.

I was wondering if there were any updates or advice on a specific version to revert to for this bug. It is messing up DNS for us so it's really affecting our usage.

Any help I can provide towards resolution?

KAM

#4 Updated by Mike Nichols almost 3 years ago

Having done some more research it appears the problem has previously come up in FreeBSD, fixed and then come back in 11.1. It's now due to be fixed in 11.2. The following is from Kristrof Provost (see https://lists.freebsd.org/pipermail/freebsd-net/2015-March/041770.html):

I think it’s likely https://svnweb.freebsd.org/base?view=revision&revision=324996
There was an issue with the fast path for IPv6 forwarding, which broke the pf fragment handling.
The patch in r324996 fixes this. It’s also been merged back to stable/11, so it’ll be part of the upcoming 11.2 release.
I can’t help you with building a patched pfSense version, but perhaps the pfSense people can once you point them at this commit.
Regards,
Kristof

#5 Updated by Johannes Petrick over 2 years ago

a possible hint:
Could it be a pf firewalling problem in handling ICMP?

While disabling pf via pfctl -d the traffic flows fine

tcpdump pf enabled:

10:57:07.955974 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 1240) 2001:1234:1234:80::1 > sipconnect.sipgate.de: [icmp6 sum ok] ICMP6, packet too big, mtu 1500

tcpdump pf disabled:

10:50:23.607406 IP6 (flowlabel 0x930b2, hlim 63, next-header Fragment (44) payload length: 1448) 2001:1234:1234::5060 > sipconnect.sipgate.de: frag (0x112e280e:0|1440) 5080 > sip: SIP, length: 1432

In- and Outbound ICMP traffic is permitted/allowed in Firewall Ruleset
pfsense version: 2.4.3-RELEASE (amd64)

#6 Updated by Mike Nichols over 2 years ago

Johannes - thanks for you comments.

AFAIK pf is an integral part of FreeBSD so we still have to wait for the FreeBSD folks to incorporate the fix in the next release (11.2 scheduled for later in 2018). For a live firewall switching off pf doesn't sound a very good idea but I don't doubt it makes the problem of dropped packets go away. I suspect that is why the pfsense team have ignored this issue to date; its not for them to fix it.

Mike

#7 Updated by Kevin A McGrail over 2 years ago

Mike Nichols wrote:

Johannes - thanks for you comments.

AFAIK pf is an integral part of FreeBSD so we still have to wait for the FreeBSD folks to incorporate the fix in the next release (11.2 scheduled for later in 2018). For a live firewall switching off pf doesn't sound a very good idea but I don't doubt it makes the problem of dropped packets go away. I suspect that is why the pfsense team have ignored this issue to date; its not for them to fix it.

Mike

ALL, I'm really sorry I didn't update this note but after significant research, this is a known bug in FreeBSD 11 and is fixed in the next release. pfSense 2.3.5-p1 is the last based on 10.3 which does not have the issue. Thanks to Mike Nichols, Samuel Abramson, Vernon Schryver, Nathan Hill, Kristof Provost & Glenn Gilley who helped diagnose the issue which effectively is really really messed up fragmentation of packets with IPv6 and UDP where the packets will disappear.

In short, the problem is in pfsense (https://svnweb.freebsd.org/base?view=revision&revision=324996) pending the release of FreeBSD 11.2 which will then lead to a new pfsense unless they want to regress back to 10.X.

Regards,
KAM

#8 Updated by Mike Nichols about 2 years ago

Update - 25th September 2018 - applied upgrade to pfSense 2.4.4 which is built on FreeBSD v11.2. Confirmed that the original problem (SIP phone does not ring on incoming call) now resolved.

Just to be clear this was always a problem with FreeBSD and not pfSense.

Mike

#9 Updated by Renato Botelho about 2 years ago

  • Status changed from New to Resolved

Fixed in 2.4.4 as reported by original submitter

#10 Updated by Luiz Souza about 2 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF