Project

General

Profile

Actions

Bug #11575

closed

OpenVPN clients cannot pass traffic when reconnecting using the same source port

Added by Jim Pingle about 3 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Viktor Gurov
Category:
OpenVPN Client Export
Target version:
-
Start date:
02/28/2021
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
Affected Plus Version:
Affected Architecture:

Description

If an OpenVPN client reconnects immediately after disconnecting, in certain cases it cannot pass traffic.

This appears to be related to the client using the same port when reconnecting. Adding nobind or lport 0 to the client configuration allows it to function as expected. This is viable for remote access clients but not all site-to-site connections which may need to use the same source port in certain cases. We do already default to lport 0 for clients on the firewall.

Initial testing doesn't show it as related to pf states or similar, since clearing the states does not affect this condition.

Since it seems to be a problem in OpenVPN itself, this is mostly for tracking and checking future versions of OpenVPN to see if they solve the problem.


Related issues

Has duplicate Todo #11574: Add "nobind" to exported OpenVPN configurations by defaultDuplicate02/28/2021

Actions
Actions #1

Updated by Jim Pingle about 3 years ago

  • Subject changed from OpenVPN clients cannot pass traffic when reconnecting to OpenVPN clients cannot pass traffic when reconnecting using the same source port

OpenVPN 2.5.1 does not appear to make a difference for this. I built a package for FreeBSD and loaded it, as well as updating a client, and the same behavior happened there.

Actions #2

Updated by IT Support about 3 years ago

adding nobind fixes the problems with viscosity on mac big sur not reconnecting after a disconnect. It continues to work after 60 seconds as well, which was another issue where it timed out.

it is under the options tab as a checkbox.

Actions #3

Updated by Edgardo Rodriguez about 3 years ago

Jim Pingle wrote:

If an OpenVPN client reconnects immediately after disconnecting, in certain cases it cannot pass traffic.

This appears to be related to the client using the same port when reconnecting. Adding nobind or lport 0 to the client configuration allows it to function as expected. This is viable for remote access clients but not all site-to-site connections which may need to use the same source port in certain cases. We do already default to lport 0 for clients on the firewall.

Initial testing doesn't show it as related to pf states or similar, since clearing the states does not affect this condition.

Since it seems to be a problem in OpenVPN itself, this is mostly for tracking and checking future versions of OpenVPN to see if they solve the problem.

I am also hit by this.
This is what better describes my scenario: https://community.openvpn.net/openvpn/ticket/1388
Enabling debug this is what I see on reconnect:
client/1.1.1.11:1194 Key [AF_INET]1.1.1.11:1194 [0] not initialized (yet), dropping packet.

I am sure by now that this is not directly related to any PFSense firewalling or any other custom thing, this is indeed related to OpenVPN

Adding lport 0 is a valid workaround (src ip:port changes and so client is "brand new"), but for short term. I would be great if renegotiation can occur without a full exchange.

Actions #4

Updated by Edgardo Rodriguez about 3 years ago

I found that, using tcp server mode reconnection works as expected (without needing to set lport 0, or nobind, or any additional parameters on clients configs)
Also, rolled back to pfSense 2.4.4 with OpenVPN 2.4.6 and this works as expected despite the transport used. It doesn't even matters if connecting frpm OpenVPN Client 2.5.1 => OpenVPN Server 2.4.6.

I suspect about enable_async_push being now enabled at compilation time, as follows:

[2.5.0-RELEASE]
OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb 5 2021
library versions: OpenSSL 1.1.1i-freebsd 8 Dec 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <>
Compile time defines: enable_async_push=yes enable_comp_stub=no enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=yes enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

[2.4.4-RELEASE]
OpenVPN 2.4.6 amd64-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Oct 3 2018
library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=yes enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=no with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

Actions #5

Updated by Edgardo Rodriguez about 3 years ago

Well, confirmed what I stated before,
enable_async_push=yes breaks reconnect process when using server with UDP as transport.
Added in this commit: https://github.com/pfsense/FreeBSD-ports/commit/076aa6a7375a5867e52c98b40ce163ef0f382c5e

So, I Updated to 2.5.1 from official FreeBSD 12's repository, which shows that this flag is turned off at compilation time, like OpenVPN defaults.
https://freebsd.pkgs.org/12/freebsd-aarch64/openvpn-2.5.1.txz.html
Steps (temporal, until pfSense package gets updated) were as follows:
(dependency included: pkg: Missing dependency 'easy-rsa')
pkg add https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/easy-rsa-3.0.8.txz
pkg add -f https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/openvpn-2.5.1.txz

pkg add https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/easy-rsa-3.0.8.txz
Fetching easy-rsa-3.0.8.txz: 100%   44 KiB  45.4kB/s    00:01
Installing easy-rsa-3.0.8...
Extracting easy-rsa-3.0.8: 100%

pkg add -f https://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/All/openvpn-2.5.1.txz
Fetching openvpn-2.5.1.txz: 100%  530 KiB 542.9kB/s    00:01
Installing openvpn-2.5.1...
package openvpn is already installed, forced install
Extracting openvpn-2.5.1: 100%
=====
Message from openvpn-2.5.1:

--
Edit /etc/rc.conf[.local] to start OpenVPN automatically at system
  startup. See /usr/local/etc/rc.d/openvpn for details.

  Connect to VPN server as a client with this command to include
  the client.up/down scripts in the initialization:
  openvpn-client <spec>.ovpn

  For compatibility notes when interoperating with older OpenVPN
  versions, please see <http://openvpn.net/relnotes.html>

  Note that OpenVPN does not officially support LibreSSL.

openvpn --version
OpenVPN 2.5.1 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb 26 2021
library versions: OpenSSL 1.1.1i-freebsd  8 Dec 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=yes enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=no with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

Restarted service and proceed with testing the exact same way as before, multiple reconnects (either from client or from server having explicit-exit-notify 1), and reconnection process goes through flawlessly (only 2 icmp echo lost).

I conclude then that this flag, for openvpn pkg, should be left in it's default value which is no.

Actions #6

Updated by Pippin MMD about 3 years ago

Asked on #openvpn-devel, this patch should fix this ticket:
https://patchwork.openvpn.net/patch/1550/

It is not related to: "enable_async_push=yes"

Actions #7

Updated by Edgardo Rodriguez about 3 years ago

Pippin MMD wrote:

Asked on #openvpn-devel, this patch should fix this ticket:
https://patchwork.openvpn.net/patch/1550/

It is not related to: "enable_async_push=yes"

Why do you think, that my problem is automatically solved then? I don't get it

Actions #8

Updated by Wesley Lucio dos Santos about 3 years ago

Pippin MMD wrote:

Asked on #openvpn-devel, this patch should fix this ticket:
https://patchwork.openvpn.net/patch/1550/

It is not related to: "enable_async_push=yes"

Pippin MMD wrote:

Asked on #openvpn-devel, this patch should fix this ticket:
https://patchwork.openvpn.net/patch/1550/

It is not related to: "enable_async_push=yes"

Hi guys!!

Is this fix scheduled to be released in the next update?

https://redmine.pfsense.org/versions/61

Actions #9

Updated by Jim Pingle about 3 years ago

  • Target version set to CE-Next

We haven't evaluated that patch yet, but it's unlikely to make it into the next release this late in the process. If OpenVPN puts out a new patch release with it in the next couple days, perhaps we could consider that, but even that's not ideal since we wouldn't have a lot of time to ensure it is stable.

Actions #10

Updated by Edgardo Rodriguez about 3 years ago

Jim Pingle wrote:

We haven't evaluated that patch yet, but it's unlikely to make it into the next release this late in the process. If OpenVPN puts out a new patch release with it in the next couple days, perhaps we could consider that, but even that's not ideal since we wouldn't have a lot of time to ensure it is stable.

Jim, do you have any clue of why I've got this issue solved by installing a version with enable_async_push disabled ? WITHOUT the patch we are discussing here? Thanks

Actions #11

Updated by Jim Pingle about 3 years ago

No, but since you compiled it on a different system and nobody else had replicated it, it's unlikely to be related without additional confirmation.

Actions #12

Updated by Edgardo Rodriguez about 3 years ago

Jim Pingle wrote:

No, but since you compiled it on a different system and nobody else had replicated it, it's unlikely to be related without additional confirmation.

I did not actually compiled it, as I mentioned below I just switched to official FreeBSD package! The only difference though, beside this flag left as default, is the version switch 2.5.0 > 2.5.1 which has a few fixes on it.

If it were possible for me I would compile my self the same pkg with the flag switched off. Why do we have on pfSense async_push enabled?

Actions #13

Updated by Yuran Yastreb about 3 years ago

Edgardo Rodriguez wrote:

Jim Pingle wrote:

No, but since you compiled it on a different system and nobody else had replicated it, it's unlikely to be related without additional confirmation.

I did not actually compiled it, as I mentioned below I just switched to official FreeBSD package! The only difference though, beside this flag left as default, is the version switch 2.5.0 > 2.5.1 which has a few fixes on it.

If it were possible for me I would compile my self the same pkg with the flag switched off. Why do we have on pfSense async_push enabled?

I installed the packages from the repository (with your example), but I still have this problem. So don't be so categorical.

Actions #14

Updated by Edgardo Rodriguez about 3 years ago

Yuran Yastreb wrote:

Edgardo Rodriguez wrote:

Jim Pingle wrote:

No, but since you compiled it on a different system and nobody else had replicated it, it's unlikely to be related without additional confirmation.

I did not actually compiled it, as I mentioned below I just switched to official FreeBSD package! The only difference though, beside this flag left as default, is the version switch 2.5.0 > 2.5.1 which has a few fixes on it.

If it were possible for me I would compile my self the same pkg with the flag switched off. Why do we have on pfSense async_push enabled?

I installed the packages from the repository (with your example), but I still have this problem. So don't be so categorical.

Hi, English is not my born language, I am sorry If anything I said sounded rude or "categorical" as you stated.
I am trying to understand, how that patch can make things better.

I did actually solved my issue by doing what I said in comment 5. But I have no concrete answer of why, just saw that difference. What I can tell is that my issue just began after 2.4.5>2.5.0 migration (of PFsense).
Can you tell us which is your current scenario? So we can share experiences and compare evenly?

Thanks and once again, sorry If anybody got offended.

Thanks!

Actions #15

Updated by Jason B almost 3 years ago

I can confirm that after upgrading our Netgate XG-7100 from 2.4.5p1 to 21.02.1 this issue began.

Neither the OpenVPN clients (nor the viscosity ones for our Mac users) had not been upgraded, however I connect the same OpenVPN clients to a pfSense box running 2.4.5 this problem does not occur.

This only seems to happen once we have upgraded to 21.02.1.

The workround of adding the following to the clients does fix the issue:
lport 0
explicit-exit-notify

Thanks

J

Actions #16

Updated by Harm V almost 3 years ago

Combined with the duplicate-cn option, this problem is actually pretty bad. (At least I suspect we're having the same problem)

When upgrading pfsense (Netgate SG-2220) from 2.4.5-RELEASE-p1 to 2.5.1, our vpn clients in the field no longer have reliable connections over openvpn.
The connection succeeds, but even ping to the pfsense (using the openvpn tunnel ip) is very flaky. ping times vary extremely (pfsense is 10.140.0.1 on its tunnel interface ovpns1)
basically no system can make a reliable TCP connection over the tunnel.

64 bytes from 10.140.0.1: icmp_seq=353 ttl=64 time=1936 ms
64 bytes from 10.140.0.1: icmp_seq=355 ttl=64 time=30229 ms
64 bytes from 10.140.0.1: icmp_seq=357 ttl=64 time=28215 ms
64 bytes from 10.140.0.1: icmp_seq=368 ttl=64 time=17217 ms
64 bytes from 10.140.0.1: icmp_seq=372 ttl=64 time=13217 ms
64 bytes from 10.140.0.1: icmp_seq=373 ttl=64 time=12218 ms
64 bytes from 10.140.0.1: icmp_seq=382 ttl=64 time=3218 ms
64 bytes from 10.140.0.1: icmp_seq=389 ttl=64 time=27.6 ms

expected is that these are all well below <10ms

The openvpn client will eventually detect it has a bad connection, and re-establish the connection... with the same result.

I cant apply (or even test) any workarounds in the vpn client config. As.... we only can access those clients via openvpn.

(our vpn clients were not upgraded or modified configuration)

setup: hundreds of clients connected using duplicate-cn (a single shared certificate)

Any idea for a workaround on the server side (pfsense)?
I tried: lport 0 and/or explicit-exit-notify 3 in the push options, with no result

We had to revert back to 2.4.5-RELEASE-p1 to get things working.

Actions #17

Updated by Marco Conca almost 3 years ago

I had the same problem.
To replicate I connect a client, then kill the openvpn.exe process.
On the pfsense the user seems connected (it didn't send the exit notification).
After that if I try to connect, no traffic reaches the pfsense or anything else in that network.

Adding the "lport 0" or "nobind" to the client solved for me

Actions #18

Updated by Viktor Gurov over 2 years ago

Actions #19

Updated by Jim Pingle over 2 years ago

  • Project changed from pfSense to pfSense Packages
  • Category changed from OpenVPN to OpenVPN Client Export
  • Status changed from New to Pull Request Review
  • Assignee set to Viktor Gurov
  • Target version deleted (CE-Next)
  • Release Notes deleted (Default)

Since this is a client problem, and clients on pfSense already get nobind or lport 0 at appropriate times, moving this to the export package seems best.

Actions #20

Updated by Viktor Gurov over 2 years ago

  • Status changed from Pull Request Review to Feedback

Merged

Actions #21

Updated by Viktor Gurov about 2 years ago

  • Has duplicate Todo #11574: Add "nobind" to exported OpenVPN configurations by default added
Actions #22

Updated by aleksei prokofiev almost 2 years ago

Checked on 22.01 release and openvpn-client-export 1.6_4 version. The nobind is presented in exported config by default.

Actions #23

Updated by Viktor Gurov almost 2 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF