pfSense

Here's some more context:

Feb 12 14:16:27 gw php: /etc/rc.packages: The command '/usr/local/etc/rc.d/haproxy.sh stop' returned exit code '1', the output was 'Stopping haproxy. Stopping haproxy. No matching processes were found'
Feb 12 14:16:27 gw kernel: done.
Feb 12 14:16:36 gw check_reload_status: Syncing firewall
Feb 12 14:17:08 gw syslogd: exiting on signal 15
Feb 12 14:17:50 gw syslogd: kernel boot file is /boot/kernel/kernel
Feb 12 14:17:54 gw check_reload_status: Syncing firewall
Feb 12 14:17:54 gw php: /etc/rc.packages: Successfully installed package: haproxy.
Feb 12 14:17:54 gw kernel: done.

So, the first line would be the old haproxy version uninstall. Then a firewall is resynced, then syslogd gets a SIGTERM, firewall is resynced again on package new version install, and then the package install completes and syslogd is gone. Hmmmm... not exactly good.

EDIT: Again the same thing with upgrade to 0.52_6:

$ ps ax | grep 'syslogd'
72518  -  Ss       0:37.46 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -l /tmp/haproxy_chroot/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf -b 10.0.0.254

After upgrade:

$ ps ax | grep 'syslogd'

System log:

Feb 14 14:58:58 gw php: /etc/rc.packages: The command '/usr/local/etc/rc.d/haproxy.sh stop' returned exit code '1', the output was 'Stopping haproxy. Stopping haproxy. No matching processes were found'
Feb 14 14:59:05 gw check_reload_status: Syncing firewall
Feb 14 14:59:34 gw syslogd: exiting on signal 15
Feb 14 14:59:34 gw syslogd: kernel boot file is /boot/kernel/kernel
Feb 14 15:00:07 gw syslogd: kernel boot file is /boot/kernel/kernel
Feb 14 15:00:12 gw check_reload_status: Syncing firewall
Feb 14 15:00:12 gw php: /etc/rc.packages: Successfully installed package: haproxy.

2.3.3/2.3.4, really anything after adding the chrooted socket commit to syslogd.

https://redmine.pfsense.org/projects/pfsense/repository/revisions/ab31acb937792bdedef11fcdbd1d98ad126ebc0b
https://redmine.pfsense.org/issues/4898

Worked on this a bit more but still haven't nailed down the reason. It is executing the command to run syslog, it's just not staying up/running. No errors, debug mode or foreground mode for syslog shows nothing at all in the output. If I run the same command by hand, it works.

This seems vaguely familiar, similar to what we saw in years past with sigprocmask and ntp but toggling that made no difference either.

A snippet of the truss output just after syslogd was run:

58519: read(9,"<118>Mar  7 12:56:28 snmpd[58486"...,1023) = 237 (0xed)
58519: sigprocmask(SIG_BLOCK,{ SIGHUP|SIGALRM },{ }) = 0 (0x0)
58519: sigprocmask(SIG_SETMASK,{ },{ SIGHUP|SIGALRM }) = 0 (0x0)
58519: sigprocmask(SIG_BLOCK,{ SIGHUP|SIGALRM },{ }) = 0 (0x0)
58519: sigprocmask(SIG_SETMASK,{ },{ SIGHUP|SIGALRM }) = 0 (0x0)
58519: sigprocmask(SIG_BLOCK,{ SIGHUP|SIGALRM },{ }) = 0 (0x0)
58519: sigprocmask(SIG_SETMASK,{ },{ SIGHUP|SIGALRM }) = 0 (0x0)
58519: read(9,0x7fffffffe150,1023)         ERR#35 'Resource temporarily unavailable'
58519: select(10,{ 4 5 6 9 },0x0,0x0,{ 0.000000 }) = 0 (0x0)
58519: kill(58406,SIGALRM)             = 0 (0x0)
58406: fork()                     = 58519 (0xe497)
58406: SIGNAL 14 (SIGALRM)
58406: process killed, signal = 14
58273: fork()                     = 58406 (0xe426)
58273: write(1,"58406\n",6)             = 6 (0x6)
58273: exit(0x0)                
58273: process exit, rval = 0
56273: read(9,"58406\n",8192)             = 6 (0x6)
56273: read(9,0x80eeef810,8192)             = 0 (0x0)

Some of that is nohup finishing up running the command and passing back the PID.

Erm, guys, this is super-annoying. Lost logging on many boxes on recent HAproxy updates without realizing what happened, something needs to be done here.

Can https://github.com/pfsense/pfsense/pull/3567 be merged to 2.3.x as well? That at least makes it possible to add the syslogd service to service watchdog. On that note, I'd say the logging is so critical feature that it should get respawned on its own when enabled without configuring anything anywhere in case syslogd gets killed for whatever reason, be it this bug or whatever other reason.

With 3a0df77eebf27d027d512a61dcbf80adefd630c4 I can now install/remove haproxy and tinc without affecting syslogd.

I did also have to remove a redundant line from tinc's install procedure as well:
https://github.com/pfsense/FreeBSD-ports/commit/b2877187d9d3582a0718b766126d79a70ff4b1d5

I'll let it soak on 2.4 a bit and if it proves OK there we can bring all of this (and that syslogd service stuff) to 2.3.

On the latest 2.4 snapshot I can install and uninstall tinc and haproxy and syslogd never dies, and it appears to reload the config OK as I can see it opening and closing the log files as expected.

Now to see how viable the same fixes are for 2.3.x.

Seems much better. Reinstalled haproxy a couple of times and syslogd keeps logging. Still wondering what's the real bug here, huh...