Project

General

Profile

Bug #4723

Can't forward UDP fragmented packets with scrubbing enabled.

Added by Dominic Blais almost 2 years ago. Updated about 1 month ago.

Status:
Feedback
Priority:
Normal
Category:
Operating System
Target version:
Start date:
05/21/2015
Due date:
% Done:

0%

Affected version:
2.2.x
Affected Architecture:
All

Description

I have a use case where I couldn't forward UDP fragmented packets thru a site to site OpenVPN tunnel. The issue isn't linked to OpenVPN itself but solely to the fact that scrubbing is applied on outgoing traffic.

Here's what happen:

1) Fragments are entering LAN interface
2) Scrub applies and fragments are reassembled
3) The firewall can process the reassembled packet
4) The kernel fragments the packet again so it can escape thru any other interface... (vpn, opt, wan..etc..)
5) The scrub out reassemble the packet
6) The packet is too big to escape the interface so it is dropped.

I managed to fix this bug by replacing "scrub on" by "scrub in on" in /etc/inc/filter.inc. Anyway, is there a need (beside random-id) to do scrubbing for outgoing traffic? Maybe it could be possible to disable scrub out when it's not TCP? Any other idea?

I think this bug wasn't present on 2.0.0.

Thank you!

VM-network.png (9.68 KB) Phillip Davis, 07/09/2015 06:43 AM

History

#1 Updated by Dominic Blais almost 2 years ago

Just thought that random-id will apply to all packets incoming another interface (LAN..etc..) prior to exit WAN. So, if there's no need to randomize PFSense's own packets "scrub in on" could be used...

#2 Updated by JayD - almost 2 years ago

I can somewhat confirm this with the following scenario:

  • Central Office
    • OVPN Server (TCP, AES-256-CBC, LZO, PSK, IF:ovpns2)
    • SNMP management console in connected LAN/DMZ
    • Version 2.2.2
  • Office Branch
    • OVPN Client
    • SNMP agent (listening on 161/udp)
    • Version 2.1.5

Connecting to the WebUI, ssh or any TCP related services on the branch firewall from a LAN connected to the central office works just fine.

However, when the SNMP management console (or any host within a connected LAN) tries to connect to the SNMP agent via 161/udp packets seem to get scrubbed when returning from the branch firewall through the ovpn interface (ovpns2):

[2.2.2-RELEASE][root@central]/root: tcpdump -nettti pflog0 host X.X.X.X
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes

00:00:28.658768 rule 9..16777216/0(match): block in on ovpns2: IP0 bad-len 0
00:00:01.003378 rule 9..16777216/0(match): block in on ovpns2: IP14 bad-len 0
00:00:00.999763 rule 9..16777216/0(match): block in on ovpns2: IP9 bad-len 0

The ovpns2 interfaces is assigned its own logical OPT interface. Now, when creating a firewall rule on that interface explicitly allowing return traffic to pass from the SNMP agent ip address to the SNMP management console everything works as expected.

#3 Updated by Phillip Davis almost 2 years ago

Maybe my situation is also related to this in some way. We do not get big ping (or I guess other big packets) from branch office LAN to Main office LAN across an OpenVPN site-to-site link. I posted recently in the forum:
https://forum.pfsense.org/index.php?topic=96043.0
That post was a bit of a wall of text, so understandably no response :)

I have recreated in a (mostly) VM situation, as per the attached diagram. The pfSense-A VM 10.49.35.37/22 is sitting on our real LAN. Behind that is pfSense-B real VM and behind that my VirtualBox host-only network with my laptop 192.168.56.1 to pfSense-B LAN 192.168.56.2 pfSense-A VM has hybrid NAT and I have told it to NAT 192.168.56.0/24 out on WAN so that I can ping from my laptop 192.168.56.1 through all this to devices on 10.49.32.0/22 and it gets NATed out and the real things like 10.49.32.9 can answer easily.

With pings up to 1472 bytes I get good packets back and forth:
From Client-B 192.168.56.1
ping 10.49.32.9 -l 1472 -t

pfSense-B OpenVPN packet capture:
17:08:08.257718 IP 192.168.56.1 > 10.49.32.9: ICMP echo request, id 1, seq 15517, length 1480
17:08:08.260448 IP 10.49.32.9 > 192.168.56.1: ICMP echo reply, id 1, seq 15517, length 1480

pfSense-A OpenVPN packet capture:
17:08:54.247187 IP 192.168.56.1 > 10.49.32.9: ICMP echo request, id 1, seq 15562, length 1480
17:08:54.248338 IP 10.49.32.9 > 192.168.56.1: ICMP echo reply, id 1, seq 15562, length 1480

pfSense-A front-side packet capture:
17:06:33.947095 IP 10.49.35.37 > 10.49.32.9: ICMP echo request, id 50743, seq 15422, length 1480
17:06:33.948408 IP 10.49.32.9 > 10.49.35.37: ICMP echo reply, id 50743, seq 15422, length 1480

Increase the ping to 1473 bytes and it has to be fragmented. This is the result:
ping 10.49.32.1 -l 1473 -t

pfSense-B OpenVPN packet capture:
17:10:49.271944 IP 192.168.56.1 > 10.49.32.9: ICMP echo request, id 1, seq 15677, length 1480
17:10:49.271966 IP 192.168.56.1 > 10.49.32.9: ip-proto-1
17:10:49.274387 IP 172.16.0.1 > 192.168.56.1: ICMP host 10.49.32.9 unreachable, length 36

pfSense-A OpenVPN packet capture:
17:09:45.039423 IP 192.168.56.1 > 10.49.32.9: ICMP echo request, id 1, seq 15610, length 1480
17:09:45.039486 IP 172.16.0.1 > 192.168.56.1: ICMP host 10.49.32.9 unreachable, length 36
17:09:45.039517 IP 192.168.56.1 > 10.49.32.9: ip-proto-1

pfSense-A front-side packet capture:
17:10:23.128146 IP 10.49.35.37 > 10.49.32.9: ICMP echo request, id 50743, seq 15650, length 1481

2 problems here:
a) pfSense-A OpenVPN sends an "unreachable" response back to 192.168.56.1 - seemingly before it even receives the 2nd fragment to reassemble.

b) pfSense-A front side at least claims to send a 1481 byte payload packet out to 10.49.32.9 - why would it even send anything onwards if it has sent an "unreachable" response? And then an over-size packet has been sent out, I guess because "scrub on WAN fragment reassemble" is in effect.

Maybe if this test environment example is sorted out, then other things will also start working?

I will try some variations with "scrub in on WAN" ...

#4 Updated by Chris Buechler over 1 year ago

  • Description updated (diff)
  • Category changed from Unknown to Operating System
  • Status changed from New to Confirmed
  • Assignee set to Chris Buechler
  • Target version set to 2.2.5
  • Affected version changed from 2.2.2 to 2.2.x

I hit this issue with a customer last week. Worked fine after disabling scrub. I have pcaps from their traffic (UBZ-69510) and should be able to replicate.

#5 Updated by Chris Buechler over 1 year ago

  • Target version changed from 2.2.5 to 2.3

#6 Updated by Jim Thompson about 1 year ago

  • Assignee changed from Chris Buechler to Luiz Otavio O Souza
  • Priority changed from High to Normal

#7 Updated by Luiz Otavio O Souza about 1 year ago

  • Target version changed from 2.3 to 2.3.1

#8 Updated by Chris Buechler about 1 year ago

  • Target version changed from 2.3.1 to 2.3.2

#9 Updated by Chris Buechler 10 months ago

  • Target version changed from 2.3.2 to 2.4.0

#10 Updated by Remko Lodder 9 months ago

Chris Buechler wrote:

I hit this issue with a customer last week. Worked fine after disabling scrub. I have pcaps from their traffic (UBZ-69510) and should be able to replicate.

I see similiar issues with:

GIF interface over IPSEC (mtu1280), reading the pflog0 output it constantly states 'bad-len', and any packet going out on the internet (mostly TCP btw) return a : ICMP Unreachable notice when the Syn/Ack comes back.

#11 Updated by Dominic Blais 6 months ago

Remko Lodder wrote:

Chris Buechler wrote:

I hit this issue with a customer last week. Worked fine after disabling scrub. I have pcaps from their traffic (UBZ-69510) and should be able to replicate.

I see similiar issues with:

GIF interface over IPSEC (mtu1280), reading the pflog0 output it constantly states 'bad-len', and any packet going out on the internet (mostly TCP btw) return a : ICMP Unreachable notice when the Syn/Ack comes back.

I think the fix is quite simple, just replace the two lines in /etc/inc/filter.inc that starts with "scrub on" by "scrub in on". Just add the "in" and everything will be fine for every case... I haven't found a reason to keep it scrubing out..

#12 Updated by Luiz Otavio O Souza 6 months ago

Remko Lodder wrote:

Chris Buechler wrote:

I hit this issue with a customer last week. Worked fine after disabling scrub. I have pcaps from their traffic (UBZ-69510) and should be able to replicate.

I see similiar issues with:

GIF interface over IPSEC (mtu1280), reading the pflog0 output it constantly states 'bad-len', and any packet going out on the internet (mostly TCP btw) return a : ICMP Unreachable notice when the Syn/Ack comes back.

I fixed the issue with pflog0 output (corrupted output in tcpdump - 'bad-len').

That was a different issue, but it is now fixed in 2.4.

#13 Updated by Luiz Otavio O Souza 6 months ago

  • Status changed from Confirmed to Feedback

I tested the forwarding of fragmented ICMP and UDP packets and they seem to be working as expected on 2.4.

Could someone else who is affected by this issue confirm that ?

Thanks!

PS: 2.4 is almost reaching beta state, so beware...

#14 Updated by Richard Gate about 1 month ago

Hi, I've hit this problem with UDP packets for RADIUS authentication when using a pfSense IPSec tunnel from an AP doing 802.1X EAP/TLS.
I found that applying the changes to /etc/inc/filter.inc, as described by Dominic Blais (many thanks for that by the way), fixed the problem.
I just wondered if there was an official pfSense patch available for this.
Thanks to all.

#15 Updated by Luiz Otavio O Souza about 1 month ago

Richard Gate wrote:

Hi, I've hit this problem with UDP packets for RADIUS authentication when using a pfSense IPSec tunnel from an AP doing 802.1X EAP/TLS.
I found that applying the changes to /etc/inc/filter.inc, as described by Dominic Blais (many thanks for that by the way), fixed the problem.
I just wondered if there was an official pfSense patch available for this.
Thanks to all.

Richard, which pfSense version are you running ?

#16 Updated by Richard Gate about 1 month ago

Luiz Otavio O Souza wrote:

Richard, which pfSense version are you running ?

Latest 2.3.3_1

Also available in: Atom PDF