Bug #13233
open
OpenVPN DCO connection fails with Auth Digest Algorithm set to SHA512
Added by Steve Wilson over 2 years ago.
Updated over 2 years ago.
Affected Plus Version:
22.05
Description
OpenVPN DCO configurations specifying an auth digest algorithm of SHA512 fail to connect. Changing the algorithm to SHA256 resolves the issue. See https://forum.netgate.com/topic/172479/openvpn-with-dco/6. It's not clear to me if this is intended (but as yet undocumented) behavior or a true bug. If DCO currently requires the auth digest algorithm to be SHA256 it should probably be flagged in the comments on the OpenVPN Server and Client set-up pages.
- Status changed from New to Feedback
Tested on 22.05.b.20220524.0600.
I was unable to reproduce this issue using OpenVPN RA TLS+User auth. Taking an existing working configuration with and without DCO enabled, I changed the Auth digest algorithm option from SHA256 (256-bit) to SHA512 (512-bit) on both the server and client, and was able to connect and reach resources behind the firewall.
Please provide specific reproducible steps.
Hopefully this will be reproducible:
1. Set up Non-DCO OpenVPN server and client with follwing config options: peer to peer, tun, udp, username + password user authentication, tls key (encryption and authentication), cipher AES-256-GCM, auth digest SHA512, subnet topology.
2. After confirming OpenVPN connects, change server and client to DCO while leaving the auth digest set to SHA512.
3. Client then fails to connect.
4. Change server and client auth digest to SHA256.
5. Client then connects without issue.
I just tried changing a working OpenVPN DCO server and client from SHA256 auth digest to SHA512 and the OpenVPN server crashed and refused to restart. Also failed to restart after a reboot. Had to roll back to a prior ZFS snapshot to recover. I would normally think that this is a 'corner case' related to some peculiarity in my setup but because at least one other user experienced the same issue (see forum post referenced above) it seems like something more is involved.
Thanks for pointing out the RA-only restriction. I see that stephenw10 has replied in the original forum string that "DCO is SHA256 only right now" so that also answers my question. Perhaps it would be best to automatically force SHA256 when the DCO checkbox is selected (much like the way the cipher is forced to AES-256-GCM) to prevent confusion - at least until SHA512 is supported.
That isn't what the P2P limitation is. The GUI selection for "peer-to-peer SSL/TLS" is fine, it's OpenVPN's internal peer-to-peer mode that breaks, which happens if you put in a tunnel network with a /30 or smaller CIDR mask (e.g. /31) tunnel network.
Also we need the whole OpenVPN config -- every single setting, not just the procedure you used. Ideally, the config.xml section for the VPN AND the /var/etc/openvpn/<instance>/config.ovpn file. You can remove private info like TLS keys and passwords but any IP addresses should stay or at least be masked consistently so we can tell different addresses apart.
- Subject changed from OpenVPN DCO Connection Fails If Auth Digest Algorithm Is Set To SHA512 to OpenVPN DCO connection fails with Auth Digest Algorithm set to SHA512
We have tested internally here and can't reproduce any problems with SHA384 or SHA512. In each case so long as both sides match they can connect and pass traffic. So there must be some other combination of settings that brings out the problem. Continue the discussion on the forum thread and report back here once a definitive diagnosis has been reached.
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by