pfSense

I've had repeated trouble with upgrades and even non-upgrade-related reboots with pkg configuration. 2.3.5 through 2.4.2, CE (on both a PC and a Protectli FW6A) and a Netgate build (on a SG-4860).

Similar symptoms to this ticket, however no ZFS involved:

https://redmine.pfsense.org/issues/6906

pkg and pkg-static return no output, GUI can't report upgrade or package status/lists.

symlinks were a recursive mess, I don't have all the details but for example:

file /var/db/pkg/pkg
/var/db/pkg/pkg: broken symbolic link to ../../root/var/db/pkg

/root/var/db/pkg was also a symlink to ../../root/var/db/pkg, similar issues for /var/cache/pkg and /root/var/cache/pkg.

Boxes would sometimes hang on reboot, needed to Ctrl-C on console.

At least one occurrence was confounded by this PHP upgrade issue, which occurred after a successful upgrade to 2.4.4:

https://www.netgate.com/docs/pfsense/install/upgrade-guide.html#version-specific-notes
https://forum.netgate.com/topic/135868/php-errors-after-upgrade-to-2-4-4

I've disabled using RAM for /var and /tmp recently and have had no further issues, however these are production devices and I'm going to test by observing normal operation (upgrades and reboots) over time.

To reproduce:

Configure /var to live in memory. Perform "normal" upgrades and reboots.

I realize this isn't a easy to repro ticket and I don't have a proposed code fix, I'm mainly filing in case others have observed similar behavior. My guess is there's some sort of race with /var in RAM, on non-ZFS systems. I looked at pfSense-rc a little but don't see anything obvious.