Bug #11541
open
OpenVPN status does not work properly when set to TCP and Concurrent Connections = 1
0%
Description
Since updating from 2.4.5 to 2.5 I am having an issue with OpenVPN when using "Peer to Peer (SSL/TLS)" mode.
Network access between the two devices (PfSense and Mikrotik) is working properly and I can ping/access devices on either network via the connection, the Mikrotik device admin interface is showing as being connected but the pfSense OpenVPN status page shows no devices are connected. (Screenshot 1)
I have a client specific override setup, which defines which certificate to use, along with some custom routes. (Screenshot 2)
Strangely, I have other OpenVPN connections setup but using the "Remote Access (User Auth)" mode, these work properly and the connected clients are shown in the status page as expected. (Screenshot 3)
The logs indicate a connection being initialised and not showing any errors. (Screenshot 4)
Files
Updated by Ryan Fitton over 3 years ago
Sorry, mistyped the screenshots.
Screenshot 1: OpenVPN Peer to Peer config settings
Screenshot 2: List of openvpn servers setup
Screenshot 3: OpenVPN status
Screenshot 4: OpenVPN log
Updated by Jim Pingle over 3 years ago
- Status changed from New to Feedback
- Target version set to CE-Next
Last time something like this happened the status output changed formats slightly for one reason or another.
It's more likely the difference in TCP vs UDP at play here than the server mode, but we need a little more info to say for sure, including:
- Are there any custom options defined in that instance which might be altering the behavior of the management interface?
- Can you try changing it from TCP to UDP to see if the status works that way?
- Can you try changing it from Peer to Peer to Remote Access to see if the status works that way? (it can still operate for site-to-site clients in that mode)
Updated by Viktor Gurov over 3 years ago
Unable to reproduce
TCP/UDP modes, Shared Key / SSL/TLS - I can always see the client connection on the Status / OpenVPN page
Updated by Ryan Fitton over 3 years ago
- File pfsense_2.4.5_tcp_client.png pfsense_2.4.5_tcp_client.png added
- File pfsense_2.4.5_tcp_client_config.png pfsense_2.4.5_tcp_client_config.png added
- File pfsense_2.4.5_udp_client.png pfsense_2.4.5_udp_client.png added
- File pfsense_2.5_tcp_server.png pfsense_2.5_tcp_server.png added
- File pfsense_2.5_udp_server.png pfsense_2.5_udp_server.png added
Hello,
Thankyou for both your quick replies.
In regards to your questions:- "Are there any custom options defined in that instance which might be altering the behavior of the management interface?": The only custom options are route pushes. Shown in my configuration screenshot.
- "Can you try changing it from TCP to UDP to see if the status works that way?": Yes, the status now works if connecting via 'Peer to Peer (SSL/TLS)' with UDP.
- "Can you try changing it from Peer to Peer to Remote Access to see if the status works that way? (it can still operate for site-to-site clients in that mode)": I have not tried this, as changing from TCP to UDP now fixes the issue.
As mentioned your advice of changing from TCP to UDP has fixed the issue. The client is now shown in the connection list of the server. I have setup a second pfSense device as a client using the same configuration details as what was in my Mikrotik router, as Mikrotik does not support using UDP for OpenVPN connections.
Screenshots:
"pfsense_2.4.5_tcp_client_config.png" - pfSense 2.4.5 client device configuration for TCP mode
"pfsense_2.4.5_tcp_client.png" - pfSense 2.4.5 client device connected via TCP
"pfsense_2.5_tcp_server.png" - pfSense 2.5 server showing no devices connected via TCP
"pfsense_2.4.5_udp_client.png" - pfSense 2.4.5 client device connected via UDP
"pfsense_2.5_udp_server.png" - pfSense 2.5 server showing device is connected via UDP
Updated by Jim Pingle over 3 years ago
There may be some specific value in your OpenVPN status output tripping it up but debugging that is a little trickier.
What we'd need to see is the info from the OpenVPN management interface in both the TCP and UDP cases to compare.
First find the ID of that particular server, which is the number at the end of its interface. For example if it's server ID 2 it would be ovpns2. Then connect to the management socket in that server's configuration directory and run the status commands:
nc -U /var/etc/openvpn/server2/sock status 1 status 2 status 3 quit
Repeat that in both TCP and UDP mode and attach the output here.
Updated by Ryan Fitton over 3 years ago
I can confirm the system location for this server is, /var/etc/openvpn/server2/. Based on the commands you sent; the outputs are below.
Main difference, looks to be in TCP mode a '>CLIENT:ENV' are listed, in UDP mode these are not listed.
Some personal info has been redacted with 'XXXXXXXXXX'.
TCP:
[2.5.0-RELEASE][admin@pfSense.localdomain]/root: nc -U /var/etc/openvpn/server2/sock >INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info >CLIENT:ESTABLISHED,2 >CLIENT:ENV,n_clients=1 >CLIENT:ENV,time_unix=1614422167 >CLIENT:ENV,time_ascii=2021-02-27 10:36:07 >CLIENT:ENV,ifconfig_pool_netmask=255.255.255.0 >CLIENT:ENV,ifconfig_pool_remote_ip=172.16.0.2 >CLIENT:ENV,trusted_port=5873 >CLIENT:ENV,trusted_ip=10.0.2.40 >CLIENT:ENV,common_name=OpenVPN User Certificate: mikrotik >CLIENT:ENV,IV_TCPNL=1 >CLIENT:ENV,IV_COMP_STUBv2=1 >CLIENT:ENV,IV_COMP_STUB=1 >CLIENT:ENV,IV_LZO=1 >CLIENT:ENV,IV_LZ4v2=1 >CLIENT:ENV,IV_LZ4=1 >CLIENT:ENV,IV_NCP=2 >CLIENT:ENV,IV_PROTO=2 >CLIENT:ENV,IV_PLAT=freebsd >CLIENT:ENV,IV_VER=2.4.9 >CLIENT:ENV,untrusted_port=5873 >CLIENT:ENV,untrusted_ip=10.0.2.40 >CLIENT:ENV,tls_serial_hex_0=03 >CLIENT:ENV,tls_serial_0=3 >CLIENT:ENV,tls_digest_sha256_0=81:40:9a:a6:a7:62:0c:b4:a7:b1:3b:91:2d:32:92:91:bd:e5:b4:32:c1:03:86:e6:60:1d:e2:35:26:b2:58:6b >CLIENT:ENV,tls_digest_0=8b:0a:10:38:57:ea:82:b4:31:cf:8b:a9:c2:52:62:23:51:6b:54:79 >CLIENT:ENV,tls_id_0=CN=OpenVPN User Certificate: mikrotik, C=GB, ST=XXXXXXXXXX, L=XXXXXXXXXX, O=pfSense.localdomain >CLIENT:ENV,X509_0_O=pfSense.localdomain >CLIENT:ENV,X509_0_L=XXXXXXXXXX >CLIENT:ENV,X509_0_ST=XXXXXXXXXX >CLIENT:ENV,X509_0_C=GB >CLIENT:ENV,X509_0_CN=OpenVPN User Certificate: mikrotik >CLIENT:ENV,tls_serial_hex_1=00 >CLIENT:ENV,tls_serial_1=0 >CLIENT:ENV,tls_digest_sha256_1=ac:3e:fa:d4:d9:52:5d:7f:86:2e:86:d6:c2:8a:39:47:89:14:ae:d3:ec:33:69:8e:a1:65:b7:3d:14:3e:35:63 >CLIENT:ENV,tls_digest_1=72:b1:20:68:bb:4a:97:fe:47:97:76:6e:3a:67:f2:8a:fc:20:aa:8c >CLIENT:ENV,tls_id_1=C=GB, ST=XXXXXXXXXX, L=XXXXXXXXXX, O=pfSense.localdomain, emailAddress=XXXXXXXXXX, CN=Open on Certificate >CLIENT:ENV,X509_1_CN=Open on Certificate >CLIENT:ENV,X509_1_emailAddress=XXXXXXXXXX >CLIENT:ENV,X509_1_O=pfSense.localdomain >CLIENT:ENV,X509_1_L=XXXXXXXXXX >CLIENT:ENV,X509_1_ST=XXXXXXXXXX >CLIENT:ENV,X509_1_C=GB >CLIENT:ENV,remote_port_1=1194 >CLIENT:ENV,local_port_1=1195 >CLIENT:ENV,local_1=XXXXXXXXXX >CLIENT:ENV,proto_1=tcp4-server >CLIENT:ENV,daemon_pid=42670 >CLIENT:ENV,daemon_start_time=1614422027 >CLIENT:ENV,daemon_log_redirect=0 >CLIENT:ENV,daemon=1 >CLIENT:ENV,verb=1 >CLIENT:ENV,config=/var/etc/openvpn/server2/config.ovpn >CLIENT:ENV,ifconfig_local=172.16.0.1 >CLIENT:ENV,ifconfig_netmask=255.255.255.0 >CLIENT:ENV,route_net_gateway=172.16.13.171 >CLIENT:ENV,route_vpn_gateway=172.16.0.2 >CLIENT:ENV,route_network_1=192.168.100.0 >CLIENT:ENV,route_netmask_1=255.255.255.0 >CLIENT:ENV,route_gateway_1=172.16.0.2 >CLIENT:ENV,route_network_2=192.168.200.0 >CLIENT:ENV,route_netmask_2=255.255.255.0 >CLIENT:ENV,route_gateway_2=172.16.0.2 >CLIENT:ENV,script_context=init >CLIENT:ENV,tun_mtu=1500 >CLIENT:ENV,link_mtu=1623 >CLIENT:ENV,dev=ovpns2 >CLIENT:ENV,dev_type=tun >CLIENT:ENV,script_type=up >CLIENT:ENV,redirect_gateway=0 >CLIENT:ENV,END status 1 OpenVPN CLIENT LIST Updated,2021-02-27 10:36:49 Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since OpenVPN User Certificate: mikrotik,10.0.2.40:5873,4779,5365,2021-02-27 10:36:07 ROUTING TABLE Virtual Address,Common Name,Real Address,Last Ref 172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08 192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08 192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08 GLOBAL STATS Max bcast/mcast queue length,0 END status 2 TITLE,OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb 5 2021 TIME,2021-02-27 10:36:53,1614422213 HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher CLIENT_LIST,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,172.16.0.2,,4851,5439,2021-02-27 10:36:07,1614422167,UNDEF,2,0,AES-256-CBC HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t) ROUTING_TABLE,172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08,1614422168 ROUTING_TABLE,192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08,1614422168 ROUTING_TABLE,192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08,1614422168 GLOBAL_STATS,Max bcast/mcast queue length,0 END status 3 TITLE OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb 5 2021 TIME 2021-02-27 10:36:56 1614422216 HEADER CLIENT_LIST Common Name Real Address Virtual Address Virtual IPv6 Address Bytes Received Bytes Sent Connected Since Connected Since (time_t) Username Client ID Peer ID Data Channel Cipher CLIENT_LIST OpenVPN User Certificate: mikrotik 10.0.2.40:5873 172.16.0.2 4851 5439 2021-02-27 10:36:07 1614422167 UNDEF 2 0 AES-256-CBC HEADER ROUTING_TABLE Virtual Address Common Name Real Address Last Ref Last Ref (time_t) ROUTING_TABLE 172.16.0.2 OpenVPN User Certificate: mikrotik 10.0.2.40:5873 2021-02-27 10:36:08 1614422168 ROUTING_TABLE 192.168.100.0/24 OpenVPN User Certificate: mikrotik 10.0.2.40:5873 2021-02-27 10:36:08 1614422168 ROUTING_TABLE 192.168.200.0/24 OpenVPN User Certificate: mikrotik 10.0.2.40:5873 2021-02-27 10:36:08 1614422168 GLOBAL_STATS Max bcast/mcast queue length 0 END quit
UDP:
[2.5.0-RELEASE][admin@pfSense.localdomain]/root: nc -U /var/etc/openvpn/server2/sock >INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info status 1 OpenVPN CLIENT LIST Updated,2021-02-27 10:43:07 Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since OpenVPN User Certificate: mikrotik,10.0.2.40:52996,5668,5616,2021-02-27 10:41:54 ROUTING TABLE Virtual Address,Common Name,Real Address,Last Ref 192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54 192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54 172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54 GLOBAL STATS Max bcast/mcast queue length,0 END status 2 TITLE,OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb 5 2021 TIME,2021-02-27 10:43:10,1614422590 HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher CLIENT_LIST,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,172.16.0.2,,5668,5616,2021-02-27 10:41:54,1614422514,UNDEF,0,0,AES-256-CBC HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t) ROUTING_TABLE,192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54,1614422514 ROUTING_TABLE,192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54,1614422514 ROUTING_TABLE,172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54,1614422514 GLOBAL_STATS,Max bcast/mcast queue length,0 END status 3 TITLE OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb 5 2021 TIME 2021-02-27 10:43:15 1614422595 HEADER CLIENT_LIST Common Name Real Address Virtual Address Virtual IPv6 Address Bytes Received Bytes Sent Connected Since Connected Since (time_t) Username Client ID Peer ID Data Channel Cipher CLIENT_LIST OpenVPN User Certificate: mikrotik 10.0.2.40:52996 172.16.0.2 5668 5688 2021-02-27 10:41:54 1614422514 UNDEF 0 0 AES-256-CBC HEADER ROUTING_TABLE Virtual Address Common Name Real Address Last Ref Last Ref (time_t) ROUTING_TABLE 192.168.200.0/24 OpenVPN User Certificate: mikrotik 10.0.2.40:52996 2021-02-27 10:41:54 1614422514 ROUTING_TABLE 192.168.100.0/24 OpenVPN User Certificate: mikrotik 10.0.2.40:52996 2021-02-27 10:41:54 1614422514 ROUTING_TABLE 172.16.0.2 OpenVPN User Certificate: mikrotik 10.0.2.40:52996 2021-02-27 10:41:54 1614422514 GLOBAL_STATS Max bcast/mcast queue length 0 END quit
Updated by Ryan Fitton over 3 years ago
Also, I should mention when running 'nc -U /var/etc/openvpn/server2/sock' in TCP mode; it takes up to 1 minute for the command to output anything.
In UDP mode, the command instantly outputs "INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info" on screen.
Updated by Viktor Gurov over 3 years ago
Ryan Fitton wrote:
Also, I should mention when running 'nc -U /var/etc/openvpn/server2/sock' in TCP mode; it takes up to 1 minute for the command to output anything.
it looks like concurrent socket access - if you run 'nc -U /var/etc/openvpn/server2/sock' and open the Status / OpenVPN page, it shows an empty table.
Have you seen the same issue after rebooting the appliance?
Updated by Ryan Fitton over 3 years ago
Yes, still the same result when the system has had a full reboot.
I've also installed a fresh copy of pfSense 2.5 on a virtual machine in case something went wrong whilst upgrading versions originally, restored my pfSense config and same result.
Updated by Jim Pingle over 3 years ago
I don't see any significant differences in the status output contents other than the TCP version you printed has a lot of extra environment info in the output. I don't see that extra info or the delay on my local TCP instances.
Not sure about the delay but as others noted it may be due to multiple processes attempting concurrent access.
Updated by Ryan Fitton over 3 years ago
I've found that if I set the 'Concurrent connections' value to anything greater than 1, my client is now shown in the OpenVPN status page.
Updated by Jim Pingle over 3 years ago
Not that I'd expect that to cause a problem, but why would you set that to 1? It doesn't make much sense.
If you don't have "Duplicate Connections" set the remote client could never connect more than one instance anyhow.
Updated by Jim Pingle over 3 years ago
- Subject changed from Connected client not shown in OpenVPN Status page when Peer to Peer (SSL/TLS) to OpenVPN status does not work properly when set to TCP and Concurrent Connections = 1
- Status changed from Feedback to New
I can replicate that here now even on Remote Access (not P2P) so it appears to be a limitation in OpenVPN itself when set to TCP with max-clients 1.
I don't know if it's considering the management query another client or what, but since it's a limitation in OpenVPN itself the best we could do is work around it.
We could have the status page and widget print a message saying it's unavailable with that combination of settings if the tunnel is set to a TCP mode with concurrent connections = 1, and maybe add a warning to the concurrent connections help text.
Updated by Ryan Fitton over 3 years ago
Jim Pingle wrote:
Not that I'd expect that to cause a problem, but why would you set that to 1? It doesn't make much sense.
If you don't have "Duplicate Connections" set the remote client could never connect more than one instance anyhow.
No real reason really. Probably something I set a while ago and never got round to changing it.