Bug #16968
open
IPsec VTI max-mss Clamping Does Not Rewrite Returning TCP SYN-ACK Packets
0%
Description
When max-mss is configured on an IPsec VTI interface, MSS clamping is applied only to outbound TCP SYN packets. Returning TCP SYN-ACK packets retain their original MSS value and are forwarded to the LAN client unchanged.
The same configuration applied to a regular routed interface clamps MSS in both directions, including the returning SYN-ACK packet.
To verify the behavior, I performed lab testing using both a regular routed interface and an IPsec VTI interface with identical max-mss settings. Packet captures were collected on the transit interface (WAN/VTI) as well as the LAN interface to observe the packet values before and after forwarding.
On the regular routed interface, the outbound SYN was rewritten as expected, and the returning SYN-ACK was also rewritten before being forwarded to the LAN client.
On the IPsec VTI interface, the outbound SYN was rewritten correctly, but the returning SYN-ACK retained its original MSS value and was forwarded to the LAN client unchanged. This behavior was consistently reproduced during testing.
Files
Updated by xavier Lemaire 21 days ago
I have been investigating this issue for several months as an intermittent TCP connectivity problem affecting printing traffic over an IPsec VTI tunnel.
After comparing my packet captures with the behavior described in issue #16968, it appears that this ticket may correspond to the problem I am experiencing.
The tested connection was:
- Client: 192.168.xx.206
- Remote printer/server: 192.168.yy.220
- TCP ports tested: 50001 and 9100
- IPsec VTI interface: ipsec2
- Packet captures were taken simultaneously on the LAN and VTI interfaces at both sites.
During the initial test on June 29, 2026, the following behavior was observed:
1. On the client-side LAN interface (SJ ix0.90), the outbound TCP SYN advertised MSS 1460.
2. On the client-side VTI (SJ ipsec2), the same SYN had been clamped to MSS 1320.
3. It arrived on the remote VTI (CO ipsec2) with MSS 1320.
4. The remote host returned a TCP SYN-ACK advertising MSS 1460.
5. This SYN-ACK remained at MSS 1460 on CO ipsec2, SJ ipsec2, and finally on the client-side LAN interface.
Capture point Outbound SYN MSS Returning SYN-ACK MSS
━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━
SJ LAN ix0.90 1460 1460
─────────────── ────────────────── ───────────────────────
SJ VTI ipsec2 1320 1460
─────────────── ────────────────── ───────────────────────
CO VTI ipsec2 1320 1460
─────────────── ────────────────── ───────────────────────
CO LAN em0.10 1320 1460This shows that MSS clamping associated with the VTI rewrites the outbound SYN, but does not rewrite the returning SYN-ACK. This behavior closely matches the description of this issue #16968.
I also tested a possible workaround by applying MSS settings to the regular routed LAN/Data interfaces at both sites, in addition to the VTI interfaces.
During the subsequent captures, the returning SYN-ACK was observed with MSS 1460 on the remote LAN interface and MSS 1360 on the remote VTI. This confirms that the packet was rewritten somewhere along the forwarding path after entering through
the regular routed interface.
However, this workaround did not resolve the actual connectivity and printing problem at all. The operational issue remained present despite the different MSS value observed in the packet captures.
Therefore, the workaround should not be considered a solution or confirmation that the problem has been fixed. It only changed the MSS value visible in the captures without restoring reliable application behavior.
Based on the initial simultaneous captures, issue #16968 still appears to describe the same VTI MSS clamping behavior that I have been troubleshooting. Further investigation is required to confirm whether this behavior is the complete root
cause of the application failure or one contributing factor.
Also available in: Atom