Project

General

Profile

Actions
Copy link

Bug #15606

closed

Data transfer problems when using interface-bound states with automatic floating states for IPsec rules

Added by Craig Coonrad 6 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Marcos M
Category:
IPsec
Target version:
2.8.0
Start date:
Due date:
% Done:

100%

Estimated time:
Plus Target Version:
24.11
Release Notes:
Force Exclusion
Affected Version:
2.8.0
Affected Architecture:

Description

Version: 24.03-RELEASE (amd64)
Platform: PVE/KVM

Test environment:

linux <--> pfSense <-- (IPsec VTI) --> pfSense <--> linux

Baseline test of iperf3 and sftp (1GB file) using floating states . (linux --> linux)
Accepted connection from 10.11.11.12, port 55973
[  5] local 10.10.10.11 port 5201 connected to 10.11.11.12 port 53279
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  81.6 MBytes   684 Mbits/sec
[  5]   1.00-2.00   sec  72.6 MBytes   609 Mbits/sec
[  5]   2.00-3.00   sec  61.9 MBytes   519 Mbits/sec
[  5]   3.00-4.00   sec  63.9 MBytes   536 Mbits/sec
[  5]   4.00-5.00   sec  72.2 MBytes   605 Mbits/sec
[  5]   5.00-6.00   sec  70.1 MBytes   588 Mbits/sec
[  5]   6.00-7.00   sec  61.8 MBytes   518 Mbits/sec
[  5]   7.00-8.00   sec  66.9 MBytes   561 Mbits/sec
[  5]   8.00-9.00   sec  82.9 MBytes   695 Mbits/sec
[  5]   9.00-10.00  sec  62.3 MBytes   522 Mbits/sec
[  5]  10.00-10.00  sec   171 KBytes   494 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   696 MBytes   584 Mbits/sec                  receiver


sftp> put tstfile
Uploading tstfile to /home/ccoonrad/tstfile
tstfile                                                                                    100% 1024MB  64.9MB/s   00:15

Test switching state policy to interface with patch #15430 applied.
Accepted connection from 10.11.11.12, port 58451
[  5] local 10.10.10.11 port 5201 connected to 10.11.11.12 port 54957
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  64.5 KBytes   528 Kbits/sec
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  64.5 KBytes  52.8 Kbits/sec                  receiver

sftp transfer starts in a stalled state (eventually something times out, and the transfer completes.)
sftp> put tstfile
Uploading tstfile to /home/ccoonrad/tstfile
tstfile                                                                                      0%    0     0.0KB/s - stalled -

The pcaps for the iperf3 and sftp sessions both show the receiver sending a bunch of ACKs, followed by a bunch of TCP retransmits from the sender. 
sender: 10.11.11.12
receiver: 10.10.10.11

Files

Download all files
iperf3-interface-states.pcap (156 KB) iperf3-interface-states.pcap iperf3 pcap Craig Coonrad, 07/07/2024 08:12 PM
sftp-interface-states.pcap (160 KB) sftp-interface-states.pcap sftp pcap Craig Coonrad, 07/07/2024 08:12 PM
Actions
Copy link
 #1

Updated by Jim Pingle 6 months ago

  • Subject changed from Data transfer problems with patch #15430 (Automatically use floating states for IPsec rules) to Data transfer problems when using interface-bound states with automatic floating states for IPsec rules
  • Assignee set to Marcos M
Actions
Copy link
 #2

Updated by Marcos M 6 months ago

  • Project changed from pfSense Plus to pfSense
  • Category changed from IPsec to IPsec
  • Status changed from New to Ready To Test
  • Target version set to 2.8.0
  • Plus Target Version set to 24.08

This seems to happen because of the bogus state that's created initially on the VTI, e.g.:

ipsec1 icmp 10.255.5.1:61473 -> 10.255.5.2:61473       0:0
   age 00:00:01, expires in 00:00:19, 1:0 pkts, 84:0 bytes, rule 102, allow-opts
   id: 4fab476600000000 creatorid: 3152b202
all icmp 10.255.5.1:61473 -> 10.255.5.2:61473       0:0
   age 00:00:01, expires in 00:00:09, 1:1 pkts, 84:84 bytes, rule 114
   id: 50ab476600000000 creatorid: 3152b202
   origif: enc0

Once the state expires (the one on ipsec1 for the example above), traffic matches the all state instead and succeeds.

This results in the initial stall behavior for the sftp transfer. The iperf test similarly stalls and then succeeds if left running for longer than the state expiration.

https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/1162

Somewhat oddly, testing shows this is not always an issue. For example, downloading/uploading a file via HTTPS works.

Actions
Copy link
 #3

Updated by Craig Coonrad 6 months ago

Tested Marcos' patch successfully <thumbs up>

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  3.67 GBytes   525 Mbits/sec  814             sender
[  5]   0.00-60.01  sec  3.67 GBytes   525 Mbits/sec                  receiver

sftp> put tstfile
tstfile                                                                                    100% 1024MB  66.4MB/s   00:15
Actions
Copy link
 #4

Updated by Marcos M 6 months ago

  • Status changed from Ready To Test to Feedback
  • % Done changed from 0 to 100
Actions
Copy link
 #5

Updated by Marcos M 6 months ago

  • Status changed from Feedback to Resolved
Actions
Copy link
 #6

Updated by Marcos M 6 months ago

  • Release Notes changed from Default to Force Exclusion
Actions
Copy link
 #7

Updated by Marcos M 6 months ago

  • Affected Version set to 2.8.0
Actions
Copy link
 #8

Updated by Jim Pingle 2 months ago

  • Plus Target Version changed from 24.08 to 24.11
Actions
Copy link

Also available in: Atom PDF