Project

General

Profile

Actions
Copy link

Bug #11541

open

OpenVPN status does not work properly when set to TCP and Concurrent Connections = 1

Added by Ryan Fitton about 3 years ago. Updated about 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
OpenVPN
Target version:
CE-Next
Start date:
02/25/2021
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.5.0
Affected Architecture:

Description

Since updating from 2.4.5 to 2.5 I am having an issue with OpenVPN when using "Peer to Peer (SSL/TLS)" mode.

Network access between the two devices (PfSense and Mikrotik) is working properly and I can ping/access devices on either network via the connection, the Mikrotik device admin interface is showing as being connected but the pfSense OpenVPN status page shows no devices are connected. (Screenshot 1)

I have a client specific override setup, which defines which certificate to use, along with some custom routes. (Screenshot 2)

Strangely, I have other OpenVPN connections setup but using the "Remote Access (User Auth)" mode, these work properly and the connected clients are shown in the status page as expected. (Screenshot 3)

The logs indicate a connection being initialised and not showing any errors. (Screenshot 4)

Files

Download all files
Screenshot 2.png (204 KB) Screenshot 2.png Ryan Fitton, 02/25/2021 03:40 PM
Screenshot 1.png (1.4 MB) Screenshot 1.png Ryan Fitton, 02/25/2021 03:40 PM
Screenshot 3.png (206 KB) Screenshot 3.png Ryan Fitton, 02/25/2021 03:40 PM
Screenshot 4.png (309 KB) Screenshot 4.png Ryan Fitton, 02/25/2021 03:40 PM
pfsense_2.4.5_tcp_client.png (39.5 KB) pfsense_2.4.5_tcp_client.png Ryan Fitton, 02/26/2021 10:37 AM
pfsense_2.4.5_udp_client.png (41.1 KB) pfsense_2.4.5_udp_client.png Ryan Fitton, 02/26/2021 10:37 AM
pfsense_2.4.5_tcp_client_config.png (1.53 MB) pfsense_2.4.5_tcp_client_config.png Ryan Fitton, 02/26/2021 10:37 AM
pfsense_2.5_tcp_server.png (101 KB) pfsense_2.5_tcp_server.png Ryan Fitton, 02/26/2021 10:37 AM
pfsense_2.5_udp_server.png (119 KB) pfsense_2.5_udp_server.png Ryan Fitton, 02/26/2021 10:37 AM
Actions
Copy link
 #1

Updated by Ryan Fitton about 3 years ago

Sorry, mistyped the screenshots.

Screenshot 1: OpenVPN Peer to Peer config settings
Screenshot 2: List of openvpn servers setup
Screenshot 3: OpenVPN status
Screenshot 4: OpenVPN log

Actions
Copy link
 #2

Updated by Jim Pingle about 3 years ago

  • Status changed from New to Feedback
  • Target version set to CE-Next

Last time something like this happened the status output changed formats slightly for one reason or another.

It's more likely the difference in TCP vs UDP at play here than the server mode, but we need a little more info to say for sure, including:

  • Are there any custom options defined in that instance which might be altering the behavior of the management interface?
  • Can you try changing it from TCP to UDP to see if the status works that way?
  • Can you try changing it from Peer to Peer to Remote Access to see if the status works that way? (it can still operate for site-to-site clients in that mode)
Actions
Copy link
 #3

Updated by Viktor Gurov about 3 years ago

Unable to reproduce
TCP/UDP modes, Shared Key / SSL/TLS - I can always see the client connection on the Status / OpenVPN page

Actions
Copy link
 #4

Updated by Ryan Fitton about 3 years ago

Hello,

Thankyou for both your quick replies.

In regards to your questions:
  • "Are there any custom options defined in that instance which might be altering the behavior of the management interface?": The only custom options are route pushes. Shown in my configuration screenshot.
  • "Can you try changing it from TCP to UDP to see if the status works that way?": Yes, the status now works if connecting via 'Peer to Peer (SSL/TLS)' with UDP.
  • "Can you try changing it from Peer to Peer to Remote Access to see if the status works that way? (it can still operate for site-to-site clients in that mode)": I have not tried this, as changing from TCP to UDP now fixes the issue.

As mentioned your advice of changing from TCP to UDP has fixed the issue. The client is now shown in the connection list of the server. I have setup a second pfSense device as a client using the same configuration details as what was in my Mikrotik router, as Mikrotik does not support using UDP for OpenVPN connections.

Screenshots:
"pfsense_2.4.5_tcp_client_config.png" - pfSense 2.4.5 client device configuration for TCP mode
"pfsense_2.4.5_tcp_client.png" - pfSense 2.4.5 client device connected via TCP
"pfsense_2.5_tcp_server.png" - pfSense 2.5 server showing no devices connected via TCP

"pfsense_2.4.5_udp_client.png" - pfSense 2.4.5 client device connected via UDP
"pfsense_2.5_udp_server.png" - pfSense 2.5 server showing device is connected via UDP

Actions
Copy link
 #5

Updated by Jim Pingle about 3 years ago

There may be some specific value in your OpenVPN status output tripping it up but debugging that is a little trickier.

What we'd need to see is the info from the OpenVPN management interface in both the TCP and UDP cases to compare.

First find the ID of that particular server, which is the number at the end of its interface. For example if it's server ID 2 it would be ovpns2. Then connect to the management socket in that server's configuration directory and run the status commands:

nc -U /var/etc/openvpn/server2/sock
status 1
status 2
status 3
quit

Repeat that in both TCP and UDP mode and attach the output here.

Actions
Copy link
 #6

Updated by Ryan Fitton about 3 years ago

I can confirm the system location for this server is, /var/etc/openvpn/server2/. Based on the commands you sent; the outputs are below.

Main difference, looks to be in TCP mode a '>CLIENT:ENV' are listed, in UDP mode these are not listed.

Some personal info has been redacted with 'XXXXXXXXXX'.

TCP:

[2.5.0-RELEASE][admin@pfSense.localdomain]/root: nc -U /var/etc/openvpn/server2/sock
>INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info
>CLIENT:ESTABLISHED,2
>CLIENT:ENV,n_clients=1
>CLIENT:ENV,time_unix=1614422167
>CLIENT:ENV,time_ascii=2021-02-27 10:36:07
>CLIENT:ENV,ifconfig_pool_netmask=255.255.255.0
>CLIENT:ENV,ifconfig_pool_remote_ip=172.16.0.2
>CLIENT:ENV,trusted_port=5873
>CLIENT:ENV,trusted_ip=10.0.2.40
>CLIENT:ENV,common_name=OpenVPN User Certificate: mikrotik
>CLIENT:ENV,IV_TCPNL=1
>CLIENT:ENV,IV_COMP_STUBv2=1
>CLIENT:ENV,IV_COMP_STUB=1
>CLIENT:ENV,IV_LZO=1
>CLIENT:ENV,IV_LZ4v2=1
>CLIENT:ENV,IV_LZ4=1
>CLIENT:ENV,IV_NCP=2
>CLIENT:ENV,IV_PROTO=2
>CLIENT:ENV,IV_PLAT=freebsd
>CLIENT:ENV,IV_VER=2.4.9
>CLIENT:ENV,untrusted_port=5873
>CLIENT:ENV,untrusted_ip=10.0.2.40
>CLIENT:ENV,tls_serial_hex_0=03
>CLIENT:ENV,tls_serial_0=3
>CLIENT:ENV,tls_digest_sha256_0=81:40:9a:a6:a7:62:0c:b4:a7:b1:3b:91:2d:32:92:91:bd:e5:b4:32:c1:03:86:e6:60:1d:e2:35:26:b2:58:6b
>CLIENT:ENV,tls_digest_0=8b:0a:10:38:57:ea:82:b4:31:cf:8b:a9:c2:52:62:23:51:6b:54:79
>CLIENT:ENV,tls_id_0=CN=OpenVPN User Certificate: mikrotik, C=GB, ST=XXXXXXXXXX, L=XXXXXXXXXX, O=pfSense.localdomain
>CLIENT:ENV,X509_0_O=pfSense.localdomain
>CLIENT:ENV,X509_0_L=XXXXXXXXXX
>CLIENT:ENV,X509_0_ST=XXXXXXXXXX
>CLIENT:ENV,X509_0_C=GB
>CLIENT:ENV,X509_0_CN=OpenVPN User Certificate: mikrotik
>CLIENT:ENV,tls_serial_hex_1=00
>CLIENT:ENV,tls_serial_1=0
>CLIENT:ENV,tls_digest_sha256_1=ac:3e:fa:d4:d9:52:5d:7f:86:2e:86:d6:c2:8a:39:47:89:14:ae:d3:ec:33:69:8e:a1:65:b7:3d:14:3e:35:63
>CLIENT:ENV,tls_digest_1=72:b1:20:68:bb:4a:97:fe:47:97:76:6e:3a:67:f2:8a:fc:20:aa:8c
>CLIENT:ENV,tls_id_1=C=GB, ST=XXXXXXXXXX, L=XXXXXXXXXX, O=pfSense.localdomain, emailAddress=XXXXXXXXXX, CN=Open on Certificate
>CLIENT:ENV,X509_1_CN=Open on Certificate
>CLIENT:ENV,X509_1_emailAddress=XXXXXXXXXX
>CLIENT:ENV,X509_1_O=pfSense.localdomain
>CLIENT:ENV,X509_1_L=XXXXXXXXXX
>CLIENT:ENV,X509_1_ST=XXXXXXXXXX
>CLIENT:ENV,X509_1_C=GB
>CLIENT:ENV,remote_port_1=1194
>CLIENT:ENV,local_port_1=1195
>CLIENT:ENV,local_1=XXXXXXXXXX
>CLIENT:ENV,proto_1=tcp4-server
>CLIENT:ENV,daemon_pid=42670
>CLIENT:ENV,daemon_start_time=1614422027
>CLIENT:ENV,daemon_log_redirect=0
>CLIENT:ENV,daemon=1
>CLIENT:ENV,verb=1
>CLIENT:ENV,config=/var/etc/openvpn/server2/config.ovpn
>CLIENT:ENV,ifconfig_local=172.16.0.1
>CLIENT:ENV,ifconfig_netmask=255.255.255.0
>CLIENT:ENV,route_net_gateway=172.16.13.171
>CLIENT:ENV,route_vpn_gateway=172.16.0.2
>CLIENT:ENV,route_network_1=192.168.100.0
>CLIENT:ENV,route_netmask_1=255.255.255.0
>CLIENT:ENV,route_gateway_1=172.16.0.2
>CLIENT:ENV,route_network_2=192.168.200.0
>CLIENT:ENV,route_netmask_2=255.255.255.0
>CLIENT:ENV,route_gateway_2=172.16.0.2
>CLIENT:ENV,script_context=init
>CLIENT:ENV,tun_mtu=1500
>CLIENT:ENV,link_mtu=1623
>CLIENT:ENV,dev=ovpns2
>CLIENT:ENV,dev_type=tun
>CLIENT:ENV,script_type=up
>CLIENT:ENV,redirect_gateway=0
>CLIENT:ENV,END
status 1
OpenVPN CLIENT LIST
Updated,2021-02-27 10:36:49
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
OpenVPN User Certificate: mikrotik,10.0.2.40:5873,4779,5365,2021-02-27 10:36:07
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08
192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08
192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08
GLOBAL STATS
Max bcast/mcast queue length,0
END
status 2
TITLE,OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb  5 2021
TIME,2021-02-27 10:36:53,1614422213
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher
CLIENT_LIST,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,172.16.0.2,,4851,5439,2021-02-27 10:36:07,1614422167,UNDEF,2,0,AES-256-CBC
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
ROUTING_TABLE,172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08,1614422168
ROUTING_TABLE,192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08,1614422168
ROUTING_TABLE,192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:5873,2021-02-27 10:36:08,1614422168
GLOBAL_STATS,Max bcast/mcast queue length,0
END
status 3
TITLE   OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb  5 2021
TIME    2021-02-27 10:36:56     1614422216
HEADER  CLIENT_LIST     Common Name     Real Address    Virtual Address Virtual IPv6 Address    Bytes Received  Bytes Sent      Connected Since Connected Since (time_t)        Username        Client ID       Peer ID Data Channel Cipher
CLIENT_LIST     OpenVPN User Certificate: mikrotik      10.0.2.40:5873  172.16.0.2              4851    5439    2021-02-27 10:36:07     1614422167      UNDEF   2       0       AES-256-CBC
HEADER  ROUTING_TABLE   Virtual Address Common Name     Real Address    Last Ref        Last Ref (time_t)
ROUTING_TABLE   172.16.0.2      OpenVPN User Certificate: mikrotik      10.0.2.40:5873  2021-02-27 10:36:08     1614422168
ROUTING_TABLE   192.168.100.0/24        OpenVPN User Certificate: mikrotik      10.0.2.40:5873  2021-02-27 10:36:08     1614422168
ROUTING_TABLE   192.168.200.0/24        OpenVPN User Certificate: mikrotik      10.0.2.40:5873  2021-02-27 10:36:08     1614422168
GLOBAL_STATS    Max bcast/mcast queue length    0
END
quit

UDP:

[2.5.0-RELEASE][admin@pfSense.localdomain]/root: nc -U /var/etc/openvpn/server2/sock
>INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info
status 1
OpenVPN CLIENT LIST
Updated,2021-02-27 10:43:07
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
OpenVPN User Certificate: mikrotik,10.0.2.40:52996,5668,5616,2021-02-27 10:41:54
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54
192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54
172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54
GLOBAL STATS
Max bcast/mcast queue length,0
END
status 2
TITLE,OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb  5 2021
TIME,2021-02-27 10:43:10,1614422590
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID,Data Channel Cipher
CLIENT_LIST,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,172.16.0.2,,5668,5616,2021-02-27 10:41:54,1614422514,UNDEF,0,0,AES-256-CBC
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
ROUTING_TABLE,192.168.200.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54,1614422514
ROUTING_TABLE,192.168.100.0/24,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54,1614422514
ROUTING_TABLE,172.16.0.2,OpenVPN User Certificate: mikrotik,10.0.2.40:52996,2021-02-27 10:41:54,1614422514
GLOBAL_STATS,Max bcast/mcast queue length,0
END
status 3
TITLE   OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Feb  5 2021
TIME    2021-02-27 10:43:15     1614422595
HEADER  CLIENT_LIST     Common Name     Real Address    Virtual Address Virtual IPv6 Address    Bytes Received  Bytes Sent      Connected Since Connected Since (time_t)        Username        Client ID       Peer ID Data Channel Cipher
CLIENT_LIST     OpenVPN User Certificate: mikrotik      10.0.2.40:52996 172.16.0.2              5668    5688    2021-02-27 10:41:54     1614422514      UNDEF   0       0       AES-256-CBC
HEADER  ROUTING_TABLE   Virtual Address Common Name     Real Address    Last Ref        Last Ref (time_t)
ROUTING_TABLE   192.168.200.0/24        OpenVPN User Certificate: mikrotik      10.0.2.40:52996 2021-02-27 10:41:54     1614422514
ROUTING_TABLE   192.168.100.0/24        OpenVPN User Certificate: mikrotik      10.0.2.40:52996 2021-02-27 10:41:54     1614422514
ROUTING_TABLE   172.16.0.2      OpenVPN User Certificate: mikrotik      10.0.2.40:52996 2021-02-27 10:41:54     1614422514
GLOBAL_STATS    Max bcast/mcast queue length    0
END
quit

Actions
Copy link
 #7

Updated by Ryan Fitton about 3 years ago

Also, I should mention when running 'nc -U /var/etc/openvpn/server2/sock' in TCP mode; it takes up to 1 minute for the command to output anything.

In UDP mode, the command instantly outputs "INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info" on screen.

Actions
Copy link
 #8

Updated by Viktor Gurov about 3 years ago

Ryan Fitton wrote:

Also, I should mention when running 'nc -U /var/etc/openvpn/server2/sock' in TCP mode; it takes up to 1 minute for the command to output anything.

it looks like concurrent socket access - if you run 'nc -U /var/etc/openvpn/server2/sock' and open the Status / OpenVPN page, it shows an empty table.

Have you seen the same issue after rebooting the appliance?

Actions
Copy link
 #9

Updated by Ryan Fitton about 3 years ago

Yes, still the same result when the system has had a full reboot.

I've also installed a fresh copy of pfSense 2.5 on a virtual machine in case something went wrong whilst upgrading versions originally, restored my pfSense config and same result.

Actions
Copy link
 #10

Updated by Jim Pingle about 3 years ago

I don't see any significant differences in the status output contents other than the TCP version you printed has a lot of extra environment info in the output. I don't see that extra info or the delay on my local TCP instances.

Not sure about the delay but as others noted it may be due to multiple processes attempting concurrent access.

Actions
Copy link
 #11

Updated by Ryan Fitton about 3 years ago

I've found that if I set the 'Concurrent connections' value to anything greater than 1, my client is now shown in the OpenVPN status page.

Actions
Copy link
 #12

Updated by Jim Pingle about 3 years ago

Not that I'd expect that to cause a problem, but why would you set that to 1? It doesn't make much sense.
If you don't have "Duplicate Connections" set the remote client could never connect more than one instance anyhow.

Actions
Copy link
 #13

Updated by Jim Pingle about 3 years ago

  • Subject changed from Connected client not shown in OpenVPN Status page when Peer to Peer (SSL/TLS) to OpenVPN status does not work properly when set to TCP and Concurrent Connections = 1
  • Status changed from Feedback to New

I can replicate that here now even on Remote Access (not P2P) so it appears to be a limitation in OpenVPN itself when set to TCP with max-clients 1.

I don't know if it's considering the management query another client or what, but since it's a limitation in OpenVPN itself the best we could do is work around it.

We could have the status page and widget print a message saying it's unavailable with that combination of settings if the tunnel is set to a TCP mode with concurrent connections = 1, and maybe add a warning to the concurrent connections help text.

Actions
Copy link
 #14

Updated by Ryan Fitton about 3 years ago

Jim Pingle wrote:

Not that I'd expect that to cause a problem, but why would you set that to 1? It doesn't make much sense.
If you don't have "Duplicate Connections" set the remote client could never connect more than one instance anyhow.

No real reason really. Probably something I set a while ago and never got round to changing it.

Actions
Copy link

Also available in: Atom PDF