pfSense

Hello,

I searched Redmine and did not find an existing pfSense issue matching COMMAND_SOCKET_WRITE_FAIL / COMMAND_SOCKET_CONNECTION_SHUTDOWN_FAIL under Kea DHCPv4 HA load. This is distinct from #16865, which was already patched in this environment.

Summary¶

Kea DHCPv4 on pfSense CE 2.8.1, running HA hot-standby, intermittently fails to answer control socket status-get requests during DHCP client activity.

During the same windows, Kea logs repeated COMMAND_SOCKET_WRITE_FAIL ... Broken pipe and occasional COMMAND_SOCKET_CONNECTION_SHUTDOWN_FAIL.

The DHCP service does not stop completely. Some VLANs continue to receive DHCP replies, but users report intermittent DHCP failures on other VLANs.

Environment¶

• Platform: pfSense CE 2.8.1
• OS: FreeBSD 15.0-CURRENT, pfSense RELENG_2_8_1
• Kea: 2.6.2 (tarball)
• Backend: Memfile
• HA mode: hot-standby
• Primary: pfmaster, 10.11.1.3:8765
• Standby: pfbackup, 10.11.1.4:8765
• Subnets: 27
• Kea interfaces: 27
• Multi-threading: enabled, 8 threads, queue size 64

Incident 2026-08-05 13:00-13:23¶

Users reported DHCP failures again between 13:00 and 13:23.

The once-per-minute watchdog stayed OK:

2026-08-05 13:00:00 OK host=pfmaster
...
2026-08-05 13:23:00 OK host=pfmaster

However, the 5-second monitor showed frequent control socket failures:

2026-08-05 13:00:11 KEA_STATUS empty_response timed_out=true
2026-08-05 13:00:18 KEA_STATUS empty_response timed_out=true
2026-08-05 13:00:30 KEA_STATUS empty_response timed_out=true
2026-08-05 13:00:37 KEA_STATUS empty_response timed_out=true

Per-minute status-get summary:

13:00 ok=4 bad=5
13:01 ok=8 bad=1
13:02 ok=6 bad=3
13:03 ok=5 bad=4
13:04 ok=3 bad=5
13:05 ok=6 bad=3
13:06 ok=4 bad=4
13:07 ok=6 bad=3
13:08 ok=5 bad=4
13:09 ok=6 bad=3
13:10 ok=4 bad=5
13:11 ok=6 bad=2
13:12 ok=4 bad=5
13:13 ok=5 bad=3
13:14 ok=6 bad=3
13:15 ok=6 bad=3
13:16 ok=4 bad=4
13:17 ok=6 bad=3
13:18 ok=9 bad=0
13:19 ok=8 bad=2
13:20 ok=7 bad=2
13:21 ok=9 bad=2
13:22 ok=11 bad=0
13:23 ok=11 bad=0

Kea logged 75 command socket errors in the same interval.

Examples:

Aug  5 13:00:13 pfmaster kea-dhcp4[49863]: ERROR [kea-dhcp4.commands] COMMAND_SOCKET_WRITE_FAIL Error while writing to command socket 136 : Broken pipe
Aug  5 13:03:33 pfmaster kea-dhcp4[49863]: ERROR [kea-dhcp4.commands] COMMAND_SOCKET_CONNECTION_SHUTDOWN_FAIL Encountered error Socket is not connected while trying to gracefully shutdown socket
Aug  5 13:21:32 pfmaster kea-dhcp4[49863]: ERROR [kea-dhcp4.commands] COMMAND_SOCKET_WRITE_FAIL Error while writing to command socket 136 : Broken pipe

At 13:22, Kea was restarted or reloaded:

Aug  5 13:22:05 pfmaster kea-dhcp4[13833]: WARN DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
Aug  5 13:22:06 pfmaster kea-dhcp4[13833]: WARN DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 8, queue size: 64

PID changed:

before: 49863
after:  13833

DHCP activity during the incident¶

The service was not completely down. Lease activity continued:

13:00 24
13:01 31
13:02 24
13:03 21
13:04 22
13:05 23
13:06 31
13:07 40
13:08 38
13:09 29
13:10 25
13:11 29
13:12 27
13:13 28
13:14 23
13:15 25
13:16 29
13:17 21
13:18 37
13:19 37
13:20 32
13:21 41
13:22 32
13:23 24

Some VLANs continued receiving DHCP traffic/replies:

• VLAN 40 had DHCP traffic and server replies from 13:00:00.
• VLAN 912 had DHCP traffic and server replies from 13:00:00.
• VLAN 302 traffic became visible around 13:15.

Recurrent pattern¶

Similar events were observed before:

2026-06-16 13:17
2026-06-17 13:14
2026-06-23 13:13
2026-06-24 13:09
2026-06-25 12:57
2026-06-30 13:00
2026-07-01 13:27
2026-07-08 13:00
2026-07-08 13:20
2026-08-04 08:03
2026-08-04 08:18
2026-08-05 13:00-13:21

Current state¶

After the incident:

result=0
globally-disabled=false
socket=ready
ha_in_touch=true
communication-interrupted=false
pid=13833

Kea is listening on DHCP UDP/67 and HA TCP/8765.

/conf/kea4_scripts.d/ is empty. No kea2unbound or unbound-control process was found.

Notes¶

The earlier pfSense kea2unbound.sh issue was already mitigated using the official Redmine #16865 patch. The current issue still happens without kea2unbound.sh present.

There is also a recurring warning:

DHCPSRV_MULTIPLE_RAW_SOCKETS_PER_IFACE current configuration will result in opening multiple broadcast capable sockets on some interfaces and some DHCP messages may be duplicated

Question¶

Is this a known Kea 2.6.2 / pfSense 2.8.1 issue where command socket requests can time out or break under DHCP load in HA hot-standby mode, while DHCP processing continues partially?

If yes, is there a recommended mitigation for pfSense?

• disable or tune multi-threading?
• change packet queue size?
• reduce raw socket duplication?
• change HA timings?
• avoid command socket polling while under load?
• upgrade Kea/pfSense?

Attachments¶

Sanitized attachments prepared:

• dhcpd-2026-08-05-1300-1323.log
• kea-status-burst-status-2026-08-05-1300-1323.log
• kea-watchdog-2026-08-05-1255-1330.log
• kea-dhcp4-sanitized-summary.json