Project

General

Profile

Actions
Copy link

Bug #7801

open

UDP fragments received over IPsec tunnel are not properly reassembled and forwarded

Added by Chris Linstruth about 4 years ago. Updated 1 day ago.

Status:
Feedback
Priority:
Normal
Assignee:
Viktor Gurov
Category:
IPsec
Target version:
2.6.0
Start date:
08/22/2017
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
22.01
Release Notes:
Default
Affected Version:
2.3.4_1
Affected Architecture:

Description

I used this to generate the large UDP datagrams on Host J1:
nping —udp —source-port 5060 —dest-port 5060 —data-length 2000 172.31.200.100
or
nping —udp —source-port 5060 —dest-port 5060 —data-length 2000 172.31.202.100

With pf enabled on both H and J

Into J LAN
01:12:22.571417 IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:12:22.571425 IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:12:23.599657 IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:12:23.599666 IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:12:24.603630 IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:12:24.603639 IP 172.31.201.100 > 172.31.200.100: ip-proto-17

Into IPsec on J
01:13:29.451155 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:13:30.463517 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:13:31.482213 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000

Out of IPsec on H
01:15:00.971973 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:15:01.997529 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:15:03.029548 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000

Out H LAN
01:16:09.025710 IP 172.31.201.100.5060 > 172.31.202.100.5060: UDP, length 2000
01:16:09.025715 IP 172.31.201.100 > 172.31.202.100: ip-proto-17
01:16:10.047232 IP 172.31.201.100.5060 > 172.31.202.100.5060: UDP, length 2000
01:16:10.047239 IP 172.31.201.100 > 172.31.202.100: ip-proto-17
01:16:11.066202 IP 172.31.201.100.5060 > 172.31.202.100.5060: UDP, length 2000
01:16:11.066209 IP 172.31.201.100 > 172.31.202.100: ip-proto-17

(Note that ICMP unreachables were returned by 172.31.202.100 since there is nothing
listening on udp/5060. They were received by nping. This verifies good end-to-end
connectivity.)

Disable pf on pfSense J to avoid fragment reassembly / scrubbing

With pf enabled on H and disabled on J

Into J LAN
01:21:32.108916 IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:21:32.108982 IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:21:33.132073 IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:21:33.132135 IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:21:34.153074 IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:21:34.153133 IP 172.31.201.100 > 172.31.200.100: ip-proto-17

Into IPsec on J
01:22:13.844357 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:22:13.844409 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:22:14.853298 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:22:14.853390 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:22:15.857681 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:22:15.857729 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100 > 172.31.200.100: ip-proto-17

Out of IPsec on H
01:23:08.108706 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:23:08.108729 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:23:09.109471 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:23:09.109493 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100 > 172.31.200.100: ip-proto-17
01:23:10.115393 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100.5060 > 172.31.200.100.5060: UDP, length 2000
01:23:10.115415 (authentic,confidential): SPI 0xc99a1e36: IP 172.31.201.100 > 172.31.200.100: ip-proto-17

Out H LAN
[No packets captured]

Note that removing the BINAT on IPsec on H changes this behavior. These captures
are the same (Replacing destination with 172.31.202.100 because no more BINAT):

Into J LAN
Into IPsec on J
Out of IPsec on H

But the following is captured on H LAN
01:41:02.519981 IP 172.31.201.100.5060 > 172.31.202.100.5060: UDP, length 2000
01:41:03.524742 IP 172.31.201.100.5060 > 172.31.202.100.5060: UDP, length 2000
01:41:04.528723 IP 172.31.201.100.5060 > 172.31.202.100.5060: UDP, length 2000

Note: Not fragmented according to the interface’s mtu 1500. This is similar to
what has been reported on #7779 for OpenVPN.

Upgraded J to 2.4-RC - Same behavior
Upgraded H to 2.4-RC - Same behavior

Files

pfSense+VPN copy 2.png (45.5 KB) pfSense+VPN copy 2.png Test Network Diagram Chris Linstruth, 08/22/2017 09:16 PM
Actions
Copy link
 #1

Updated by Jim Thompson about 4 years ago

  • Assignee set to Jim Pingle
Actions
Copy link
 #2

Updated by Renato Botelho about 4 years ago

  • Target version changed from 2.4.0 to 2.4.1
Actions
Copy link
 #3

Updated by Jim Pingle about 4 years ago

  • Target version changed from 2.4.1 to 2.4.2

Moving target to 2.4.2 as we need 2.4.1 sooner than anticipated.

Actions
Copy link
 #4

Updated by Jim Pingle about 4 years ago

  • Target version changed from 2.4.2 to 2.4.3
Actions
Copy link
 #5

Updated by Steve Beaver over 3 years ago

  • Assignee changed from Jim Pingle to Luiz Souza
  • Target version changed from 2.4.3 to 2.4.4

++target_version

Actions
Copy link
 #6

Updated by Franciszek Koltuniuk about 3 years ago

Hi,
I have a similar issue with fragmented packets send/received over IPsec tunnel.
Finally, I manage to update /tmp/rules.debug and make traffic to be passed as expected:

  • add scrub rule that apply scrubbing on a traffic from networks behind IPsec vpn:

scrub from <from_ipsec_network> to any no-df fragment reassemble

  • add scrub rult that disable scrubbing on a traffic from a local interface to the network behind IPsec vpn:

no scrub on $INTERFACE to <from_ipsec_network>

So the only one thing is missing: pfsense web interface that allows to setup ipsec in the same way..

Actions
Copy link
 #7

Updated by Steve Beaver about 3 years ago

  • Target version changed from 2.4.4 to 48
Actions
Copy link
 #8

Updated by Andi Admin almost 3 years ago

Any chance to get fixed soon? This bug actually prevent our VPN from being usable for VoIP which uses UDP and in some cases need fragments.

Actions
Copy link
 #9

Updated by Next Next almost 3 years ago

Hi, I also have a similar issue with fragmented packets and IPsec tunnels (noticed with ICMP traffic).
Incoming fragmented packets (either from 'LAN to IPsec' or 'IPsec to LAN') are forwarded unfragmented (packets larger then interface MTU).
Disabling scrubbing globally makes the problem go away, but this doesn't seem to be a good solution as it affects much more than only bug-related traffic.

Is there any info on what's causing this? Is it PFsense/config related or maybe something in BSD itself ?

Actions
Copy link
 #10

Updated by Gabriel Latour almost 3 years ago

Hi, I have been waiting a year for that fix, for us, it's RDS sessions that disconnects randomly when using UDP over IPSEC.

Actions
Copy link
 #11

Updated by Jim Pingle over 2 years ago

  • Target version changed from 48 to 2.5.0
Actions
Copy link
 #12

Updated by Jim Pingle about 2 years ago

See also: #9184, #7837

Actions
Copy link
 #13

Updated by Steve Beaver 12 months ago

  • Target version changed from 2.5.0 to CE-Next
Actions
Copy link
 #14

Updated by Rai Wol 8 months ago

I hit the same issue with EAP-TLS (Wireless authentication) UDP fragmented packages from AP to NPS (Radius) server not arriving. After turning off scrubbing it started working. Maybe add an option to enable/disable scrubbing per interface.

Actions
Copy link
 #15

Updated by Viktor Gurov 6 months ago

Franciszek Koltuniuk wrote:

Hi,
I have a similar issue with fragmented packets send/received over IPsec tunnel.
Finally, I manage to update /tmp/rules.debug and make traffic to be passed as expected:

  • add scrub rule that apply scrubbing on a traffic from networks behind IPsec vpn:

scrub from <from_ipsec_network> to any no-df fragment reassemble

  • add scrub rult that disable scrubbing on a traffic from a local interface to the network behind IPsec vpn:

no scrub on $INTERFACE to <from_ipsec_network>

So the only one thing is missing: pfsense web interface that allows to setup ipsec in the same way..

related to #10493:
"Because of filter_get_vpns_list() returns not only IPsec networks, IPsec MSS clamping option will affect unnecessary VPN types"

Actions
Copy link
 #17

Updated by Jim Pingle 6 months ago

  • Status changed from New to Pull Request Review
  • Target version changed from CE-Next to 2.6.0
Actions
Copy link
 #18

Updated by Renato Botelho 4 months ago

  • Status changed from Pull Request Review to Feedback
  • Assignee changed from Luiz Souza to Viktor Gurov
  • Plus Target Version set to 21.09

PR has been merged. Thanks!

Actions
Copy link
 #19

Updated by Viktor Gurov 4 months ago

  • % Done changed from 0 to 100
Actions
Copy link
 #20

Updated by Chris Linstruth 4 months ago

  • Status changed from Feedback to Assigned

The new checkboxes in System > Advanced, Firewall & NAT are not populated when re-entering the configuration page.

Checking IP Do-Not-Fragment compatibility and IP Fragment Reassemble seem to work but going back tot he config page they are unchecked again.

Enable Maximum MSS looks like it "sticks."

The oversized, fragmented packets do look like they are going from LAN host to LAN host though so that looks to be working.

Actions
Copy link
 #21

Updated by Viktor Gurov 4 months ago

Chris Linstruth wrote:

The new checkboxes in System > Advanced, Firewall & NAT are not populated when re-entering the configuration page.

Checking IP Do-Not-Fragment compatibility and IP Fragment Reassemble seem to work but going back tot he config page they are unchecked again.

Enable Maximum MSS looks like it "sticks."

fix:
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/294

Actions
Copy link
 #22

Updated by Renato Botelho 4 months ago

  • Status changed from Assigned to Feedback
Actions
Copy link
 #23

Updated by Chris Linstruth 3 months ago

OK that looks better. Thanks.

Actions
Copy link
 #24

Updated by Jim Pingle 3 months ago

  • Status changed from Feedback to Resolved
Actions
Copy link
 #25

Updated by Marcos Mendoza 2 months ago

  • Status changed from Resolved to Feedback

I was able to test this fix and noticed there are two issues which I needed to work around in order for large df-bit-set traffic to flow correctly.

  • The vpn_networks alias does not take into account IPsec P2s with an address type, only network. Hence, I needed to update the tunnels accordingly.
  • The new IP Do-Not-Fragment compatibility option cannot be unchecked after it's been checked and saved. To reproduce:
    1. Check the option, then save.
    2. Refresh the page to verify it's still checked.
    3. Uncheck the option then save.
    4. Refresh the page. The option is now still checked.
Actions
Copy link
 #26

Updated by Marcos Mendoza 2 months ago

  • Status changed from Feedback to Pull Request Review

The following merge request addresses the two issues outlined in my previous comment:
https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/356

Actions
Copy link
 #27

Updated by Jim Pingle about 2 months ago

  • Status changed from Pull Request Review to Feedback

PR merged.

Actions
Copy link
 #28

Updated by Jim Pingle 1 day ago

  • Plus Target Version changed from 21.09 to 22.01
Actions
Copy link

Also available in: Atom PDF