Long delays with LDAP enabled w/local users during boot at "Synchronizing user settings..."
- an LDAP authentication server is selected under User Manager > Settings > Authentication Server
- the selected LDAP server is configured using a FQDN rather than an IP address
- the FQDN of the LDAP server is not configured as an override in the DNS resolver and therefore is not in the /etc/hosts file.
The userlog show a delay of around 5 minutes per user:
2016-05-19 10:58:21 [unknown:usermod] root(0):wheel(0):Charlie &:/root:/bin/sh 2016-05-19 10:58:21 [unknown:useradd] admin(0):wheel(0):System Administrator:/root:/etc/rc.initial 2016-05-19 10:58:21 [unknown:useradd] admin(0) home /root made 2016-05-19 11:03:23 [unknown:useradd] xxxxx(2001):nobody(65534):X XXXX:/home/xxxxx:/sbin/nologin 2016-05-19 11:03:23 [unknown:useradd] xxxxx(2001) home /home/xxxxx made 2016-05-19 11:08:23 [unknown:useradd] yyyyy(2002):nobody(65534):Y YYYY:/home/yyyyy:/sbin/nologin 2016-05-19 11:08:23 [unknown:useradd] yyyyy(2002) home /home/yyyyy made 2016-05-19 11:13:24 [unknown:useradd] zzzzz(2003):nobody(65534):Z ZZZZ:/home/zzzzz:/sbin/nologin 2016-05-19 11:13:24 [unknown:useradd] zzzzz(2003) home /home/zzzzz made ...
This seemed to start immediately after an upgrade from 2.3.0 to 2.3.1, however I cannot see any difference in the relevant startup order between these versions so I am unsure why I haven't encountered it previously.
It would make sense however to have the DNS resolver available before any code involving DNS queries.
ie. could/should rc.bootup run these
/* start dnsmasq service */ services_dnsmasq_configure(); /* start unbound service */ services_unbound_configure();
ahead of this:
echo "Synchronizing user settings..."; local_sync_accounts();
#1 Updated by Sandeep K V over 3 years ago
Though LDAP is causing this delay, I do not think it is due to this as we have the same rc.bootup file for long time. This is my understanding - if it was what was causing the delay you must have had faced the similar scenario back before upgrade. I think it is due to LDAP timeout I believe. Let me see!
#2 Updated by Chris Buechler over 3 years ago
- Subject changed from System appears to hang on startup at "Synchronizing user settings..." to Long delays with LDAP enabled w/local users during boot at "Synchronizing user settings..."
- Category set to User Manager / Privileges
- Status changed from New to Confirmed
- Target version set to 2.3.2
This came about because I fixed #6352, so it's now doing things it should have been doing but wasn't previously in the group modification. Because your auth's set to LDAP, it tries to connect at that point when it's obtaining user permissions (though for syncing local users, that's unnecessary).
Reducing the LDAP timeout helps some, but that alone is inadequate. The loop it goes through hits that timeout multiple times per user depending on config (looks like 12 times in Brett's case), so even reducing the timeout to 5 seconds would still leave a 1 minute delay per user there with Brett's config. I went ahead and committed that though, since it does help, and 25 seconds is an excessive delay there.
2.3.1_1 will have the reduced LDAP timeout. A proper fix, to not do LDAP lookups while creating the local user database (since it's not relevant there) will require more work and testing.
#7 Updated by Ilya Kogan over 2 years ago
I hate to comment on an old issue but I couldn't find one for the "proper fix" as mentioned above. Today my LDAP server ran out of open files and, rather than simply not accept connections, connections would hang. This caused the web GUI to freeze up and, upon rebooting pfSense, would cause it to get stuck at "Synchronizing user settings...". You can imagine that it's not particularly desirable to have a working network connection and LDAP server just for a network appliance to boot.
Is this something that can be done in the background maybe and not hold up the boot process with some very strong timeouts that can withstand an LDAP server hanging the TCP connection indefinitely?