Bug #14977
closed
Stale Kea control socket lock file can prevent Kea from starting
100%
Description
If for some reason the Kea control socket lock file is present while Kea is stopped, then Kea can never start until that file is removed.
Attempting to start Kea while /tmp/kea4-ctrl-socket.lock exists, even if Kea is not currently running, results in a fatal error preventing Kea from running:
Nov 13 13:07:53 kea-dhcp4 32826 ERROR [kea-dhcp4.dhcp4.0x1c655d812000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
Removing the socket file allows Kea to start (rm /tmp/kea4-ctrl-socket.lock).
Forum thread: https://forum.netgate.com/post/1135910
Updated by Suriname Clubcard about 1 month ago
I just experienced exactly this same problem:
DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
pfSense 2.7.2.
Updated by Jim Pingle about 1 month ago
- Status changed from New to Closed
- % Done changed from 0 to 100
I can't reproduce this at all on current snapshots. I've killed the process and left the old file in place, killed it and made a file owned by another user, killed it and made a file with invalid permissions, and so on. Every time it starts right back up perfectly. I've tried restarting the process by save/apply, by using the service controls, and so on. I even killed it, left the lock in place, and rebooted. Every time it starts right up.
It's possible they fixed it upstream or something else changed that made it a non-issue.
Updated by Nicholas Ruddick about 15 hours ago
This issue was occurring for me too, and I have found a resolution. It manifests itself as a "ghost" kea process which is not logging, or visible to keactrl.
Log entries:
| May 2 18:32:00 | kea-dhcp4 | 46377 | ERROR [kea-dhcp4.dhcp4.0xbb13ee12000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable |
| May 2 18:32:00 | kea-dhcp4 | 46377 | ERROR [kea-dhcp4.dhcp4.0xbb13ee12000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable |
| May 2 18:32:00 | kea-dhcp4 | 46377 | ERROR [kea-dhcp4.dhcp4.0xbb13ee12000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable |
Ghost process not recognised by keactrl:
[2.7.2-RELEASE][root@firewall]/root: ps aux | grep kea
root 68622 0.0 0.5 52584 20556 - S 06:51 0:03.26 /usr/local/sbin/kea-dhcp4 -c /usr/local/etc/kea/kea-dhcp4.conf
root 82813 0.0 0.1 12752 2164 0 S+ 18:36 0:00.00 grep kea
[2.7.2-RELEASE][root@firewall]/tmp: ls -l /tmp/ | grep kea
srwxr-xr-x 1 root wheel 0 May 2 06:51 kea4-ctrl-socket
-rw------- 1 root wheel 0 May 2 06:51 kea4-ctrl-socket.lock
[2.7.2-RELEASE][root@firewall]/tmp: keactrl version
keactrl: 2.4.0
kea-dhcp4: 2.4.0
kea-dhcp6: 2.4.0
kea-dhcp-ddns: 2.4.0
kea-ctrl-agent: 2.4.0
[2.7.2-RELEASE][root@firewall]/root: keactrl status
DHCPv4 server: inactive
DHCPv6 server: inactive
DHCP DDNS: inactive
Control Agent: inactive
Kea DHCPv4 configuration file: /usr/local/etc/kea/kea-dhcp4.conf
Kea DHCPv6 configuration file: /usr/local/etc/kea/kea-dhcp6.conf
Kea DHCP DDNS configuration file: /usr/local/etc/kea/kea-dhcp-ddns.conf
Kea Control Agent configuration file: /usr/local/etc/kea/kea-ctrl-agent.conf
keactrl configuration file: /usr/local/etc/kea/keactrl.conf
Restarting from command line fails:
[2.7.2-RELEASE][root@firewall]/tmp: /usr/local/sbin/kea-dhcp4 -c /usr/local/etc/kea/kea-dhcp4.conf
2024-05-02 18:41:54.873 INFO [kea-dhcp4.dhcp4/5978.0x6e5c6812000] DHCP4_STARTING Kea DHCPv4 server version 2.4.0 (stable) starting
2024-05-02 18:41:54.877 ERROR [kea-dhcp4.dhcp4/5978.0x6e5c6812000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf'
: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
keactrl stop does not work:
[2.7.2-RELEASE][root@firewall]/tmp: keactrl stop
INFO/keactrl: kea-dhcp4 isn't running.
INFO/keactrl: kea-dhcp6 isn't running.
INFO/keactrl: kea-dhcp-ddns isn't running.
INFO/keactrl: kea-ctrl-agent isn't running.
[2.7.2-RELEASE][root@firewall]/tmp: ps aux | grep kea
root 68622 0.0 0.5 52584 20556 - S 06:51 0:03.30 /usr/local/sbin/kea-dhcp4 -c /usr/local/etc/kea/kea-dhcp4.conf
kill INT or -2 successfully clears the process:
[2.7.2-RELEASE][root@firewall]/tmp: kill -1 68622
[2.7.2-RELEASE][root@firewall]/tmp: ps aux | grep kea
root 68622 0.5 0.5 52584 20740 - S 06:51 0:03.35 /usr/local/sbin/kea-dhcp4 -c /usr/local/etc/kea/kea-dhcp4.conf
root 10329 0.0 0.1 12752 2168 0 S+ 18:48 0:00.00 grep kea
[2.7.2-RELEASE][root@firewall]/tmp: kill -2 68622
[2.7.2-RELEASE][root@firewall]/tmp: ps aux | grep kea
root 19501 0.0 0.1 12752 2164 0 S+ 18:49 0:00.00 grep kea
kea-dhcp4 services then starts successfully:
[2.7.2-RELEASE][root@firewall]/tmp: ps aux | grep kea
root 53145 0.0 0.5 39640 19580 - S 18:50 0:00.06 /usr/local/sbin/kea-dhcp4 -c /usr/local/etc/kea/kea-dhcp4.conf
root 99465 0.0 0.1 12752 2164 0 S+ 18:50 0:00.00 grep kea
[2.7.2-RELEASE][root@firewall]/tmp: keactrl status
DHCPv4 server: active
DHCPv6 server: inactive
DHCP DDNS: inactive
Control Agent: inactive
Kea DHCPv4 configuration file: /usr/local/etc/kea/kea-dhcp4.conf
Kea DHCPv6 configuration file: /usr/local/etc/kea/kea-dhcp6.conf
Kea DHCP DDNS configuration file: /usr/local/etc/kea/kea-dhcp-ddns.conf
Kea Control Agent configuration file: /usr/local/etc/kea/kea-ctrl-agent.conf
keactrl configuration file: /usr/local/etc/kea/keactrl.conf
BUT then oddly shuts down again exactly 1 minute later:
May 2 19:07:00 firewall kea-dhcp4[58287]: INFO [kea-dhcp4.dhcp4.0x2139c3212000] DHCP4_STARTED Kea DHCPv4 server version 2.4.0 started
May 2 19:07:54 firewall kea-dhcp4[58287]: INFO [kea-dhcp4.commands.0x2139c3212000] COMMAND_RECEIVED Received command 'lease4-get-all'
May 2 19:08:00 firewall kea-dhcp4[58287]: INFO [kea-dhcp4.dhcp4.0x2139c3212000] DHCP4_SHUTDOWN server shutdown
I checked Service Watchdog configuration and could see both dhcpd AND kea-dhcpd4 as actively monitored!
Once I REMOVED dhcpd from Service Watchdog, the repeated stop and start of the kea-dhcp4 service was resolved. So could the bug be fixed by checking to remove "dhcpd" from Service Watchdog when a user transitions dhcp configuration from ISC DHCP (Deprecated) to Kea DHCP?
Updated by Yuri Weinstein about 9 hours ago
I have experienced the same problem today 2 days after updating to 24.03-RELEASE
In addition to all described above, it seems that TailScale service was messed up as well, after the kea failure.
This bug seems alive