Bug #11626
closedGoogle LDAP connections fail due to lack of SNI for TLS 1.3
100%
Description
https://forum.netgate.com/topic/161725/google-ldap-connection-failed:
I have a problem after update my Netgate XG-7100 to the version 21.02-RELEASE-p1.
After the update my pfSense failed to bind to ldap.google.com.
Attempting connection to ldap.google.com OK
Attempting bind to ldap.google.com failed
openldap24-client doesn't support SNI:
# ldapsearch -H ldaps://ldap.google.com -x -d1 ldap_url_parse_ext(ldaps://ldap.google.com) ldap_create ldap_url_parse_ext(ldaps://ldap.google.com:636/??base) ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap.google.com:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 216.239.32.58:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 attempting to connect: connect success TLS trace: SSL_connect:before SSL initialization TLS trace: SSL_connect:SSLv3/TLS write client hello TLS trace: SSL_connect:SSLv3/TLS write client hello TLS trace: SSL_connect:SSLv3/TLS read server hello TLS trace: SSL_connect:TLSv1.3 read encrypted extensions TLS trace: SSL_connect:SSLv3/TLS read server certificate request TLS certificate verification: depth: 0, err: 18, subject: /OU=No SNI provided; please fix your client./CN=invalid2.invalid, issuer: /OU=No SNI provided; please fix your client./CN=invalid2.invalid TLS certificate verification: Error, self signed certificate TLS trace: SSL3 alert write:fatal:unknown CA TLS trace: SSL_connect:error in error TLS: can't connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate). ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
https://support.google.com/a/answer/9190869?hl=en:
Verify return code: 18 (self signed certificate) The Secure LDAP service requires a TLS client that supports and initiates a TLS session using SNI (Server Name Indication). If the TLS client does not support SNI, then the TLS server (ldap.google.com) returns a self-signed certificate that will not pass CA validation checks, to indicate that SNI is required. This behavior can be confirmed by checking the OpenSSL client output for the following line near the start of the output: depth=0 OU = "No SNI provided; please fix your client.", CN = invalid2.invalid Causes for this error may include an OpenSSL version that does not support SNI, or an application that uses the OpenSSL library with SNI explicitly disabled.
PHP issue: https://stackoverflow.com/questions/65885108/sni-issues-connecting-to-google-ldap-server
ldapsearch issue: https://www.openldap.org/lists/openldap-bugs/202002/msg00421.html
Files
Updated by Jim Pingle over 3 years ago
If OpenLDAP ldapsearch fails directly it's unlikely to be related to #9417
All the references I see to SNI seem fairly recent, but I'm not sure if they changed something on the Google side or if something on the client side changed (e.g. now using TLS 1.3 where it couldn't before).
Updated by Viktor Gurov over 3 years ago
Using the STunnel package as a workaround helps:
https://docs.netgate.com/pfsense/en/latest/recipes/auth-google-gsuite.html#install-the-stunnel-pfsense-package-ce-or-2-4-4-release
Updated by Jim Pingle over 3 years ago
- Project changed from pfSense to pfSense Plus
- Subject changed from google LDAP connection failed to Google LDAP connection failed due to lack of SNI for TLS 1.3
- Category changed from Authentication to Authentication
- Affected Version deleted (
2.5.0) - Affected Plus Version set to 21.02
Not that I like the idea of downgrading to a lower TLS version but I wonder if it would work if we forced off TLS 1.3 in this case using something like this:
ldap_set_option(NULL, LDAP_OPT_X_TLS_CIPHER_SUITE, "NORMAL:!VERS-TLS1.3");
I haven't tried that option so I'm not sure if it's one that needs NULL
in that first spot like the cert setup or if it can take $ldap
for per-connection options.
If that does work we could add a GUI knob to disable TLS 1.3 for LDAP, maybe only when sending a client cert.
Updated by Alders Watne over 3 years ago
Theoretically that would be the fix (forcing TLSv1.2 to bypass the SNI TLS v1.3 requirement), but setting this LDAP option or even with the old `putenv('LDAPTLS_CIPHER_SUITE=NORMAL:!VERS-TLS1.3')` doesn't work.
Same with `LDAPTLS_CIPHER_SUITE=NORMAL:!VERS-TLS1.3 ldapsearch -H ldaps://ldap.google.com -x -d1`.
Any ideas?
Jim Pingle wrote:
Not that I like the idea of downgrading to a lower TLS version but I wonder if it would work if we forced off TLS 1.3 in this case using something like this:
[...]
I haven't tried that option so I'm not sure if it's one that needs
NULL
in that first spot like the cert setup or if it can take$ldap
for per-connection options.If that does work we could add a GUI knob to disable TLS 1.3 for LDAP, maybe only when sending a client cert.
Updated by Jim Pingle over 3 years ago
It would either be this:
ldap_set_option(NULL, LDAP_OPT_X_TLS_CIPHER_SUITE, "NORMAL:!VERS-TLS1.3");
Or this:
ldap_set_option($ldap, LDAP_OPT_X_TLS_CIPHER_SUITE, "NORMAL:!VERS-TLS1.3");
And likely set just before or after the places LDAP_OPT_X_TLS_REQUIRE_CERT
is set in /etc/inc/auth.inc
.
Updated by Alders Watne over 3 years ago
The error self-signed error is gone but the bind still is unsuccessful. Same config ported over the 2.4 release line. Nothing specific about the error in the logs.
Jim Pingle wrote:
It would either be this:
[...]
Or this:
[...]
And likely set just before or after the places
LDAP_OPT_X_TLS_REQUIRE_CERT
is set in/etc/inc/auth.inc
.
Updated by Michael Mogren about 3 years ago
This issue open 5 months and still no fix. The stunnel workaround is not a good solution - it only works for gui and does not work for ssh login and requires additional package/configuration. Please fix.
Updated by Philipp Wagner about 3 years ago
I have the same issue when setting up Google's "Secure LDAP" as user directory and did a bit more digging. Here's my take on it.
- Server Name Indication (SNI) changed quite a bit in TLS 1.3, see also https://wiki.openssl.org/index.php/TLS1.3 and https://en.wikipedia.org/wiki/Server_Name_Indication
- pfSense before 2021.x shipped with OpenSSL < 1.1.1, a version which does not support TLS 1.3
- OpenLDAP 2.5 (a new major version in the OpenLDAP versioning scheme) added SNI support (https://bugs.openldap.org/show_bug.cgi?id=9176, https://www.openldap.org/software/release/announce.html). pfSense 21.05 (in my netgate plus version) comes with OpenLDAP 2.4.58.
So the problem that we see today is a result of an OpenSSL version that supports TLS 1.3 and a OpenLDAP version that doesn't support SNI.
The PHP code in /etc/inc/auth.inc
looks correct to me.
The real solution is to update OpenLDAP to version 2.5. I have confirmed locally that this solves the problem.
Until that happens, there are some workarounds, with of course rather different (security) properties:- Use stunnel.
- In theory: Disable server certificate validation, which then accepts the self-signed "dummy" cert that Google sends if no SNI is present. However, I don't see a way to do that in the UI, even though the server-side code has a branch for it (https://github.com/pfsense/pfsense/blob/7fb96c6c266d8086597e00053087c9bcb7281f88/src/etc/inc/auth.inc#L1008).
What does not work is setting ldap_set_option($ldap, LDAP_OPT_X_TLS_CIPHER_SUITE, "NORMAL:!VERS-TLS1.3");
. This syntax is for GnuTLS, but pfSense has built OpenLDAP with OpenSSL. OpenSSL does not seem to have a way to disable TLS 1.3 in a similar fashion as GnuTLS, see also https://mta.openssl.org/pipermail/openssl-users/2019-March/010150.html.
Finally, if you're looking for a reasonably easy to reproduce the problem with more log information take the following script, save it to your pfSense machine, and run it on the terminal (e.g. sudo php test.php
).
<?php
ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 255);
$ds=ldap_connect("ldaps://ldap.google.com:636");
# There will be certs in the directory after you've set up Google Secure LDAP in the UI. Adjust the filename.
ldap_set_option(NULL, LDAP_OPT_X_TLS_CERTFILE, "/var/run/certs/your-cert.crt");
ldap_set_option(NULL, LDAP_OPT_X_TLS_KEYFILE, "/var/run/certs/your-cert.key");
ldap_set_option($ds, LDAP_OPT_PROTOCOL_VERSION, 3);
ldap_set_option($ds, LDAP_OPT_REFERRALS, 0);
$re = ldap_start_tls($ds);
if (!$re) {
echo ldap_error($ldap);
ldap_get_option($ldap, LDAP_OPT_DIAGNOSTIC_MESSAGE, $err);
echo $err;
}
echo "connected!";
$results = ldap_search($ldap, 'dc=example,dc=com', 'uid=*');
print_r(ldap_get_entries($ldap, $results));
ldap_close($ds);
Here's how it looks for me:
$ sudo php test.php ldap_url_parse_ext(ldap://localhost/) ldap_init: trying /usr/local/etc/openldap/ldap.conf ldap_init: using /usr/local/etc/openldap/ldap.conf ldap_init: HOME env is /root ldap_init: trying /root/ldaprc ldap_init: trying /root/.ldaprc ldap_init: trying ldaprc ldap_init: LDAPCONF env is NULL ldap_init: LDAPRC env is NULL ldap_create ldap_url_parse_ext(ldaps://ldap.google.com:636) ldap_extended_operation_s ldap_extended_operation ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap.google.com:636 ldap_new_socket: 11 ldap_prepare_socket: 11 ldap_connect_to_host: Trying 216.239.32.58:636 ldap_pvt_connect: fd: 11 tm: -1 async: 0 attempting to connect: connect success TLS trace: SSL_connect:before SSL initialization TLS trace: SSL_connect:SSLv3/TLS write client hello TLS trace: SSL_connect:SSLv3/TLS write client hello TLS trace: SSL_connect:SSLv3/TLS read server hello TLS trace: SSL_connect:TLSv1.3 read encrypted extensions TLS trace: SSL_connect:SSLv3/TLS read server certificate request TLS certificate verification: depth: 0, err: 18, subject: /OU=No SNI provided; please fix your client./CN=invalid2.invalid, issuer: /OU=No SNI provided; please fix your client./CN=invalid2.invalid TLS certificate verification: Error, self signed certificate TLS trace: SSL3 alert write:fatal:unknown CA TLS trace: SSL_connect:error in error TLS: can't connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate). ldap_err2string
Updated by Jim Pingle about 3 years ago
Thanks for tracking that down!
The main problem now is that OpenLDAP 2.5 isn't currently available in the FreeBSD ports tree, which is a blocker. I'm not sure why, as it's been around for a while.
Either the OpenLDAP port maintainer will need to add it, or someone will need to submit the new set of ports for OpenLDAP 2.5 to FreeBSD.
Updated by Viktor Gurov about 3 years ago
Philipp Wagner wrote in #note-9:
- In theory: Disable server certificate validation, which then accepts the self-signed "dummy" cert that Google sends if no SNI is present. However, I don't see a way to do that in the UI, even though the server-side code has a branch for it (https://github.com/pfsense/pfsense/blob/7fb96c6c266d8086597e00053087c9bcb7281f88/src/etc/inc/auth.inc#L1008).
Does this work if you manually replace LDAP_OPT_X_TLS_HARD
with LDAP_OPT_X_TLS_NEVER
in auth.inc
?
Updated by Kris Phillips almost 3 years ago
It appears that openldap25-client and openldap25-server are both in freshports for FreeBSD.
Updated by Jim Pingle almost 3 years ago
- Assignee set to Renato Botelho
- Target version set to 22.01
Updated by Jim Pingle almost 3 years ago
- Assignee changed from Renato Botelho to Luiz Souza
Looks like the ports tree is setup to where we can define the version to use without much fuss. It may be as simple as this change before building:
diff --git a/tools/conf/pfPorts/make.conf b/tools/conf/pfPorts/make.conf
index 2f4a9bf9ca..5358cb5247 100644
--- a/tools/conf/pfPorts/make.conf
+++ b/tools/conf/pfPorts/make.conf
@@ -28,6 +28,9 @@ PHP_FD_SETSIZE= 3172
WITH_DEBUG= yes
. endif
+# Use OpenLDAP 2.5
+WITH_OPENLDAP_VER=25
+
# Options for packages
# databases
Updated by Jim Pingle over 2 years ago
- Target version changed from 22.01 to 22.05
Nudge this ahead so we have more time to ensure there aren't any regressions from the change.
Updated by Jim Pingle over 2 years ago
- Target version changed from 22.05 to 23.01
Updated by Kris Phillips over 2 years ago
Jim Pingle wrote in #note-15:
Nudge this ahead so we have more time to ensure there aren't any regressions from the change.
Jim,
Can we get this into 22.09 builds after 22.05 is out? This is a common issue people are running into more and more.
Updated by Jim Pingle over 2 years ago
Not up to me, it'll need to be handled by Luiz or Brad once things start moving for 22.09 but it's already on the radar.
Updated by Kris Phillips almost 2 years ago
Builds for 23.01 are including openldap26-client version 2.6.3, so this can probably be marked as Feedback.
Updated by Jim Pingle almost 2 years ago
- Status changed from New to Feedback
- Start date deleted (
03/05/2021) - % Done changed from 0 to 100
Updated by Jim Pingle almost 2 years ago
- Subject changed from Google LDAP connection failed due to lack of SNI for TLS 1.3 to Google LDAP connections fail due to lack of SNI for TLS 1.3
Updating subject for release notes.
Updated by Jim Pingle almost 2 years ago
- File clipboard-202212211513-cwaqi.png clipboard-202212211513-cwaqi.png added
- Status changed from Feedback to Resolved
We don't have an account with Google LDAP auth any longer, but I can confirm that the LDAP client is sending the SNI header during TLS negotiation which was the required point here.