Project

General

Profile

Actions

Regression #15152

closed

Systems with low RAM fail to upgrade to 24.03

Added by Steve Wheeler 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Operating System
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
24.03
Release Notes:
Force Exclusion
Affected Version:
Affected Architecture:
SG-1100

Description

The 1100 fails to upgrade to 24.03 with 'no space left' errors. This appears to be on any tmpfs device. So that's /var/run on a default install or /var, /tmp if ram disks are enabled.

There are new tmpfs values in FreeBSD 15 which are probably affecting this.

The 1100 has 1GB RAM but any low RAM system is likely affected. The 2100, with 4GB, upgrades no problem.

With RAM disks enabled:

[46/241] Extracting dbus-1.14.10_1,1: ........
pkg-static: Fail to create temporary directory: /var/.pkgtemp.lib.ixFrQrANvzd0:No space left on device
pkg-static: Fail to create /var/lib:No space left on device
[46/241] Extracting dbus-1.14.10_1,1... done
>>> Unlocking package pfSense-pkg-aws-wizard... done.
>>> Unlocking package pfSense-pkg-ipsec-profile-wizard... done.
>>> Unlocking package pfSense-pkg-nut... done.
/etc/pfSense-rc: /usr/local/sbin/php-fpm: not found
/etc/pfSense-rc: /etc/rc.ecl: not found
Launching the init system.../etc/pfSense-rc: /etc/rc.bootup: not found
Starting CRON... done.
/usr/local/libexec/pfSense-upgrade: php: not found
/usr/local/libexec/pfSense-upgrade: php: not found
/usr/local/libexec/pfSense-upgrade: php: not found
/usr/local/libexec/pfSense-upgrade: /usr/local/sbin/-repo-setup: not found
fcgicli: Could not connect to server(/var/run/php-fpm.socket).
/etc/rc.update_pkg_metadata: /usr/local/sbin/read_global_var: not found
Netgate pfSense Plus 23.09.1-RELEASE arm64 20231206-2022
Bootup complete

FreeBSD/arm64 (Amnesiac) (ttyu0)

login: 

Actions #1

Updated by Kristof Provost 3 months ago

I've not yet been able to reproduce this on a low-memory VM (1GB total RAM).

From the log I would guess that this case (as was the case in my test) had RAM disks enabled.

It'd be interesting to see both `df -hl` and `sysctl -a | grep tmpfs` output, because there have been a couple of changes to make tmpfs allocation stricter and it's not clear to me which one we ran into here.

Actions #2

Updated by Craig Coonrad 3 months ago

Tested 24.03 upgrade on a VM with 768MB RAM. With and without RAM Disks enabled. Unable to reproduce. Upgrades completed successfully.

Actions #3

Updated by Jim Pingle 3 months ago

I'm seeing a similar failure though so far only on 1100.

There is a sysctl oid vfs.tmpfs.memory_percent which allegedly can be set to 100 to change the tmpfs behavior back, but it can't be applied as a loader.conf.local tunable and seems to get applied too late when configured as a system tunable. It may only be possible to set it after configuring a tmpfs device so we might need to patch in a change to set that just after configuring /var/run like this, for example:

diff --git a/src/etc/pfSense-rc b/src/etc/pfSense-rc
index 2a1577a187..82095de71d 100755
--- a/src/etc/pfSense-rc
+++ b/src/etc/pfSense-rc
@@ -292,6 +292,7 @@ varrunpath=$(/usr/local/bin/php -n /usr/local/sbin/read_global_var varrun_path " 

 if ! ramdisk_is_active; then
        /sbin/mount -o rw,size=4m,mode=1777 -t tmpfs tmpfs $varrunpath
+       /sbin/sysctl vfs.tmpfs.memory_percent=100
 fi

 echo
diff --git a/src/etc/rc.ramdisk_functions.sh b/src/etc/rc.ramdisk_functions.sh
index 09c1c198ca..5dc1a010e9 100755
--- a/src/etc/rc.ramdisk_functions.sh
+++ b/src/etc/rc.ramdisk_functions.sh
@@ -131,6 +131,7 @@ ramdisk_try_mount () {
        if [ ramdisk_check_size ]; then
                SIZE=$(eval echo \${${NAME}size})m
                /sbin/mount -o rw,size=${SIZE},mode=1777 -t tmpfs tmpfs /${NAME}
+               /sbin/sysctl vfs.tmpfs.memory_percent=100
                return $?
        else
                return 1;

A different possibility is that it shouldn't be in /var/run there at all for the cwd, it makes little sense to do that, so it's not clear why it's even using space there. Changing this line in the boot script may also help that:

diff --git a/src/etc/pfSense-rc b/src/etc/pfSense-rc
index 2a1577a187..b529ca6e10 100755
--- a/src/etc/pfSense-rc
+++ b/src/etc/pfSense-rc
@@ -394,7 +394,7 @@ if [ "$DEVFS" = "0" ]; then
 fi

 # Create an initial utmp file
-cd $varrunpath && /bin/cp /dev/null utmp && /bin/chmod 644 utmp
+/bin/cp /dev/null ${varrunpath}/utmp && /bin/chmod 644 ${varrunpath}/utmp

 echo -n "." 
 /sbin/ldconfig -elf /usr/lib /usr/local/lib /lib

It's possible that code may be needed for something like running off ISO/Memstick so it can write certain temp files but it could just as easily be unnecessary legacy code from the days of embedded systems with read-only root directories.

Actions #4

Updated by Jim Pingle 3 months ago

Something else I may have narrowed down is that in each case it appears to only be dbus that's failing this. It's dbus above and dbus in my case as well. I updated just dbus and then tried the whole upgrade again and it succeeded. It's possible there is something wrong in that port's post-install script/actions that other ports aren't doing.

After the system boots I can pkg upgrade -fy dbus without error but if I force vfs.tmpfs.memory_percent=50 (much lower than the default), that fails with a similar error message as before:

[1/1] Extracting dbus-1.14.10_3,1: 100%     53 B   0.1kB/s    00:01    
ldconfig: mkstemp(/var/run/ld-elf.so.hints.LJbXPA): No space left on device
ldconfig: mkstemp(/var/run/ld-elf32.so.hints.eJ3tOx): No space left on device
pkg: POST-INSTALL script failed

That happens no matter what the current working directory is, so the cd /var/run appears to be irrelevant here.

Actions #5

Updated by Kristof Provost 3 months ago

The most recent src merge (done yesterday) includes this commit:

commit 2e68c5a44c40848c43849cc3650331cf82714dfa
Author: Mike Karels <karels@FreeBSD.org>
Date:   Tue Jan 23 11:23:38 2024 -0600

    tmpfs: increase vfs.tmpfs.memory_percent to 100 as workaround

    The changes to avoid letting tmpfs use all of memory + swap do not
    work well with ZFS ARC.  The ARC can grow quite large, and will shrink
    when there is memory pressure, but tmpfs does not allow for that.
    Pending investigation of the right way to handle this, change the
    default value of the vfs.tmpfs.memory_percent sysctl to 100 as a
    workaround.  The sysctl can be set to 95 to get back to the previous
    default.

That should fix the issues we've seen with low RAM systems.

Actions #6

Updated by Steve Wheeler 3 months ago

  • Status changed from New to Resolved

Looks good in todays snapshot:

24.03-DEVELOPMENT (arm64)
built on Mon Feb 5 6:00:00 GMT 2024
FreeBSD 15.0-CURRENT

Actions #7

Updated by Jim Pingle 3 months ago

  • Release Notes changed from Default to Force Exclusion
Actions

Also available in: Atom PDF