pfSense

Just realised the packet capture example was truncated by one character. Here's what it should look like:

16:56:59.061983 IP6 2001:ab7::5 > 2a00:23c5:d007:f800:7e2f:80ff:fe20:ce8a: frag (0|1448) 5060 > 5060: UDP, bad length 1475 > 1440
16:56:59.062024 IP6 2001:ab7::5 > 2a00:23c5:d007:f800:7e2f:80ff:fe20:ce8a: frag (1448|35)
16:56:59.062132 IP6 2a00:23c5:d007:f800:230:18ff:fec9:ce4a > 2001:ab7::5: ICMP6, packet too big, mtu 1500, length 1240

I have adjusted the PPPoE MTU to be 1500 since the previous packet capture. It's made no difference to the bug and the fragmented UDP packets are still being erroneously dropped.

Just wanted to file a METOO on this where it seems to be causing issues with IPv6 and UDP especially where we saw it with DNS and the DCC.

For example, we did some more troubleshooting on this. It looks like the firewall just simply isn’t allowing packet fragmentation at all. I sent a huge string over udp on that port and got a response. First we tried just a regular small string:

[2.2.6-RELEASE][root@XYZ.local]/root: echo -n “hi” | nc -6u dcc1.pccc.com 6277

Got a response on the firewall:

16:19:32.425288 e8:9a:8f:be:1b:9e > 00:15:5d:14:10:11, ethertype IPv6 (0x86dd), length 64: (flowlabel 0xf0bd1, hlim 48, next-header UDP (17) payload length: 10) 2600:1700:6020:2e80:6a05:caff:fe20:f59a.42821 > 2604:9100:7:9::1:33.6277: [udp sum ok] UDP, length 2

Good, previous problem of no UDP making it through is gone. But it looks small…let’s try a bigger string and try to stretch that stream across a huge packet:

(Forgive length here)

[2.2.6-RELEASE][root@XYZ.local]/root: echo -n "012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789" | nc -6u dcc1.pccc.com 6277

Came back with another response on the firewall, this time length 960. Still single packet we tried again, basically did the above string but 10x as long, should have seen 2 packets. No response at all, no breaking up into multiple fragments. either.

When we got to that point, that's when we found this bug report. We think we are seeing the same and the above information might help recreate the issue more readily.

Mike Nichols wrote:

This issue came to light when I encountered a problem with a SIP phone not receiving SIP Invite messages resulting in missed calls. This had previously worked OK and while I have not regression tested to find when it appeared I think it may be with v2.4.0 onward. Furthermore the problem can not be reproduced using pfSense v2.3.5 i386. I am running pfSense v2.4.2 using FreeBSD 11.1-RELEASE-p4 on a Mini-ITX based system with Celeron N2930 processor.

I was wondering if there were any updates or advice on a specific version to revert to for this bug. It is messing up DNS for us so it's really affecting our usage.

Any help I can provide towards resolution?

KAM

Having done some more research it appears the problem has previously come up in FreeBSD, fixed and then come back in 11.1. It's now due to be fixed in 11.2. The following is from Kristrof Provost (see https://lists.freebsd.org/pipermail/freebsd-net/2015-March/041770.html):

I think it’s likely https://svnweb.freebsd.org/base?view=revision&revision=324996
There was an issue with the fast path for IPv6 forwarding, which broke the pf fragment handling.
The patch in r324996 fixes this. It’s also been merged back to stable/11, so it’ll be part of the upcoming 11.2 release.
I can’t help you with building a patched pfSense version, but perhaps the pfSense people can once you point them at this commit.
Regards,
Kristof

a possible hint:
Could it be a pf firewalling problem in handling ICMP?

While disabling pf via pfctl -d the traffic flows fine

tcpdump pf enabled:

10:57:07.955974 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 1240) 2001:1234:1234:80::1 > sipconnect.sipgate.de: [icmp6 sum ok] ICMP6, packet too big, mtu 1500

tcpdump pf disabled:

10:50:23.607406 IP6 (flowlabel 0x930b2, hlim 63, next-header Fragment (44) payload length: 1448) 2001:1234:1234::5060 > sipconnect.sipgate.de: frag (0x112e280e:0|1440) 5080 > sip: SIP, length: 1432

In- and Outbound ICMP traffic is permitted/allowed in Firewall Ruleset
pfsense version: 2.4.3-RELEASE (amd64)

Johannes - thanks for you comments.

AFAIK pf is an integral part of FreeBSD so we still have to wait for the FreeBSD folks to incorporate the fix in the next release (11.2 scheduled for later in 2018). For a live firewall switching off pf doesn't sound a very good idea but I don't doubt it makes the problem of dropped packets go away. I suspect that is why the pfsense team have ignored this issue to date; its not for them to fix it.

Mike

Mike Nichols wrote:

Johannes - thanks for you comments.

AFAIK pf is an integral part of FreeBSD so we still have to wait for the FreeBSD folks to incorporate the fix in the next release (11.2 scheduled for later in 2018). For a live firewall switching off pf doesn't sound a very good idea but I don't doubt it makes the problem of dropped packets go away. I suspect that is why the pfsense team have ignored this issue to date; its not for them to fix it.

Mike

ALL, I'm really sorry I didn't update this note but after significant research, this is a known bug in FreeBSD 11 and is fixed in the next release. pfSense 2.3.5-p1 is the last based on 10.3 which does not have the issue. Thanks to Mike Nichols, Samuel Abramson, Vernon Schryver, Nathan Hill, Kristof Provost & Glenn Gilley who helped diagnose the issue which effectively is really really messed up fragmentation of packets with IPv6 and UDP where the packets will disappear.

In short, the problem is in pfsense (https://svnweb.freebsd.org/base?view=revision&revision=324996) pending the release of FreeBSD 11.2 which will then lead to a new pfsense unless they want to regress back to 10.X.

Regards,
KAM

Update - 25th September 2018 - applied upgrade to pfSense 2.4.4 which is built on FreeBSD v11.2. Confirmed that the original problem (SIP phone does not ring on incoming call) now resolved.

Just to be clear this was always a problem with FreeBSD and not pfSense.

Mike