pfSense

I've not yet been able to reproduce this on a low-memory VM (1GB total RAM).

From the log I would guess that this case (as was the case in my test) had RAM disks enabled.

It'd be interesting to see both `df -hl` and `sysctl -a | grep tmpfs` output, because there have been a couple of changes to make tmpfs allocation stricter and it's not clear to me which one we ran into here.

Tested 24.03 upgrade on a VM with 768MB RAM. With and without RAM Disks enabled. Unable to reproduce. Upgrades completed successfully.

I'm seeing a similar failure though so far only on 1100.

There is a sysctl oid vfs.tmpfs.memory_percent which allegedly can be set to 100 to change the tmpfs behavior back, but it can't be applied as a loader.conf.local tunable and seems to get applied too late when configured as a system tunable. It may only be possible to set it after configuring a tmpfs device so we might need to patch in a change to set that just after configuring /var/run like this, for example:

diff --git a/src/etc/pfSense-rc b/src/etc/pfSense-rc
index 2a1577a187..82095de71d 100755
--- a/src/etc/pfSense-rc
+++ b/src/etc/pfSense-rc
@@ -292,6 +292,7 @@ varrunpath=$(/usr/local/bin/php -n /usr/local/sbin/read_global_var varrun_path " 

 if ! ramdisk_is_active; then
        /sbin/mount -o rw,size=4m,mode=1777 -t tmpfs tmpfs $varrunpath
+       /sbin/sysctl vfs.tmpfs.memory_percent=100
 fi

 echo
diff --git a/src/etc/rc.ramdisk_functions.sh b/src/etc/rc.ramdisk_functions.sh
index 09c1c198ca..5dc1a010e9 100755
--- a/src/etc/rc.ramdisk_functions.sh
+++ b/src/etc/rc.ramdisk_functions.sh
@@ -131,6 +131,7 @@ ramdisk_try_mount () {
        if [ ramdisk_check_size ]; then
                SIZE=$(eval echo \${${NAME}size})m
                /sbin/mount -o rw,size=${SIZE},mode=1777 -t tmpfs tmpfs /${NAME}
+               /sbin/sysctl vfs.tmpfs.memory_percent=100
                return $?
        else
                return 1;

A different possibility is that it shouldn't be in /var/run there at all for the cwd, it makes little sense to do that, so it's not clear why it's even using space there. Changing this line in the boot script may also help that:

diff --git a/src/etc/pfSense-rc b/src/etc/pfSense-rc
index 2a1577a187..b529ca6e10 100755
--- a/src/etc/pfSense-rc
+++ b/src/etc/pfSense-rc
@@ -394,7 +394,7 @@ if [ "$DEVFS" = "0" ]; then
 fi

 # Create an initial utmp file
-cd $varrunpath && /bin/cp /dev/null utmp && /bin/chmod 644 utmp
+/bin/cp /dev/null ${varrunpath}/utmp && /bin/chmod 644 ${varrunpath}/utmp

 echo -n "." 
 /sbin/ldconfig -elf /usr/lib /usr/local/lib /lib

It's possible that code may be needed for something like running off ISO/Memstick so it can write certain temp files but it could just as easily be unnecessary legacy code from the days of embedded systems with read-only root directories.

Something else I may have narrowed down is that in each case it appears to only be dbus that's failing this. It's dbus above and dbus in my case as well. I updated just dbus and then tried the whole upgrade again and it succeeded. It's possible there is something wrong in that port's post-install script/actions that other ports aren't doing.

After the system boots I can pkg upgrade -fy dbus without error but if I force vfs.tmpfs.memory_percent=50 (much lower than the default), that fails with a similar error message as before:

[1/1] Extracting dbus-1.14.10_3,1: 100%     53 B   0.1kB/s    00:01    
ldconfig: mkstemp(/var/run/ld-elf.so.hints.LJbXPA): No space left on device
ldconfig: mkstemp(/var/run/ld-elf32.so.hints.eJ3tOx): No space left on device
pkg: POST-INSTALL script failed

That happens no matter what the current working directory is, so the cd /var/run appears to be irrelevant here.

The most recent src merge (done yesterday) includes this commit:

commit 2e68c5a44c40848c43849cc3650331cf82714dfa
Author: Mike Karels <karels@FreeBSD.org>
Date:   Tue Jan 23 11:23:38 2024 -0600

    tmpfs: increase vfs.tmpfs.memory_percent to 100 as workaround

    The changes to avoid letting tmpfs use all of memory + swap do not
    work well with ZFS ARC.  The ARC can grow quite large, and will shrink
    when there is memory pressure, but tmpfs does not allow for that.
    Pending investigation of the right way to handle this, change the
    default value of the vfs.tmpfs.memory_percent sysctl to 100 as a
    workaround.  The sysctl can be set to 95 to get back to the previous
    default.

That should fix the issues we've seen with low RAM systems.