pfSense

I've updated twice, once before submitting the bug, and again just now (the dashboard keeps saying update available, I don't know pfSense looks for updates, I have downloaded the latest from the mirror). SSHD is still not starting. My current version is:
--------
2.0-BETA3
built on Fri Jun 18 05:38:39 EDT 2010
FreeBSD pfsense1.hupfeldt 8.1-RC1 FreeBSD 8.1-RC1 #0: Fri Jun 18 05:37:34 EDT 2010 root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386 i386
--------

I initially installed, skipped the wizard and restored a config from an earlier beta. Please let me know what logs you might need. I can't see any sshd log entries in the syslog.

The update checks do not work right now, ignore that.

Then I'd need at least the output of:

# ls -l /etc/sshd
# ls -l /conf/sshd

And also:

# killall -9 sshd
# /usr/sbin/sshd -d

I did not try restoring a configuration, but technically since it's BETA we only support upgrading from 1.2.3, not from earlier BETA/ALPHA/etc releases, though it should work in 99% of cases.

I lost my long edit because I was messing with my firewall:(, so once more:

$ ls l /conf/sshd
total 4
-rw------ 1 root wheel 987 Jun 19 00:58 ssh_host_key
rw-r--r- 1 root wheel 652 Jun 19 00:58 ssh_host_key.pub

Compared to a fresh install I just made on my spare box, rsa and dsa keys are missing. On the new install I enabled ssh from the "Admin Access" tab, which seems to create these keys. When restoring a configuration with sshd enabled, it seems the keys are not created. Neither are they created if disabling/enabling sshd.

It would also be a good idea to create the keys if missing when ssh is enabled.

Lars Hupfeldt Nielsen wrote:

It would also be a good idea to create the keys if missing when ssh is enabled.

Fixing that would, by consequence, fix the actual issue, since it's really the same problem. :-)

I suspect the SSH upgrade code is not working correctly. At some point the SSH config item was moved.

What I meant about create on enabling is that it should happen every time ssh is enabled, if the keys are missing. Right now it seems that only the initial enabling creates the keys. Of course, if the code is reused for the restore scenario that is fine:)

I think there might still be an issue here, for Nanobsd at least.

Tested this with "2.0-Beta4 (I386) Built on Thu Dec 2 11:27:45 EST 2010 Nanobsd"

Updated virgin install to that version, then restored config from several months ago. After the reboot I received this error.

Dec 3 09:47:23     php: : New alert found: SSHD failed to start.
Dec 3 09:47:23     php: : The command '/usr/sbin/sshd' returned exit code '1', the output was 'Could not load host key: /etc/ssh/ssh_host_rsa_key Could not load host key: /etc/ssh/ssh_host_dsa_key Disabling protocol version 2. Could not load host key sshd: no hostkeys available -- exiting.'
Dec 3 09:47:23     php: : New alert found: pfSense has completed creating your SSH keys. SSH is now started.
Dec 3 09:46:09     php: : New alert found: pfSense has started creating your SSH keys. SSH Startup will be delayed. Please note that reloading the filter rules and changes will be delayed until this operation is completed.

ps auxw | egrep 'ss[h]'
root   63359  0.0  0.4  3316   928  ??  Is    9:47AM   0:00.02 /usr/local/sbin/sshlockout_pf

[2.0-BETA4][root@nt-firewall]/conf(14): ls -l /etc/sshd
-rwxr-xr-x  1 root  wheel  6748 Dec  2 08:31 /etc/sshd
[2.0-BETA4][root@nt-firewall]/conf(15): ls -l /conf/sshd
ls: /conf/sshd: No such file or directory

[2.0-BETA4][root@nt-firewall]/conf(16): killall -9 sshd
No matching processes were found
[2.0-BETA4][root@nt-firewall]/conf(17): /usr/sbin/sshd -d
debug1: sshd version OpenSSH_5.4p1 FreeBSD-20100308
debug1: could not open key file '/etc/ssh/ssh_host_rsa_key': No such file or directory
Could not load host key: /etc/ssh/ssh_host_rsa_key
debug1: could not open key file '/etc/ssh/ssh_host_dsa_key': No such file or directory
Could not load host key: /etc/ssh/ssh_host_dsa_key
Disabling protocol version 2. Could not load host key
sshd: no hostkeys available -- exiting.

[2.0-BETA4][root@nt-firewall]/etc/ssh(20): ls -l
total 133
-rw-r--r--  1 root  wheel  125811 Dec  2 10:12 moduli
-rw-r--r--  1 root  wheel    1683 Dec  2 10:12 ssh_config
-rw-------  1 root  wheel     992 Dec  3 09:46 ssh_host_key
-rw-r--r--  1 root  wheel     657 Dec  3 09:46 ssh_host_key.pub
-rw-r--r--  1 root  wheel     428 Dec  3 09:46 sshd_config

So something kept the system from generating the ssh_host_rsa_key files and the ssh_host_dsa_key files.

When I run /etc/sshd from the command line, it generates the keys and starts sshd.

[2.0-BETA4][root@nt-firewall]/etc/ssh(34): ls -l
total 137
-rw-r--r--  1 root  wheel  125811 Dec  2 10:12 moduli
-rw-r--r--  1 root  wheel    1683 Dec  2 10:12 ssh_config
-rw-------  1 root  wheel     668 Dec  3 10:58 ssh_host_dsa_key
-rw-r--r--  1 root  wheel     617 Dec  3 10:58 ssh_host_dsa_key.pub
-rw-------  1 root  wheel     992 Dec  3 10:58 ssh_host_key
-rw-r--r--  1 root  wheel     657 Dec  3 10:58 ssh_host_key.pub
-rw-------  1 root  wheel    1675 Dec  3 10:58 ssh_host_rsa_key
-rw-r--r--  1 root  wheel     409 Dec  3 10:58 ssh_host_rsa_key.pub
-rw-r--r--  1 root  wheel     428 Dec  3 10:58 sshd_config

[2.0-BETA4][root@nt-firewall]/etc/ssh(36): ps auxw | egrep 'ss[h]'
root   56177  0.0  1.3  5272  3160  ??  Is   10:58AM   0:00.00 /usr/sbin/sshd
root   63359  0.0  0.4  3316   928  ??  Is    9:47AM   0:00.02 /usr/local/sbin/sshlockout_pf

Is it possible that some other startup script is re-mounting the filesystem RO during the time that the keys are being
generated? The /conf/sshd dir was not created either the first time.

Do any of the other logs between the start of the regen and the end have to do with a process that re mounts the filesystem?

Dec 3 09:47:23     php: : New alert found: SSHD failed to start.
Dec 3 09:47:23     php: : The command '/usr/sbin/sshd' returned exit code '1', the output was 'Could not load host key: /etc/ssh/ssh_host_rsa_key Could not load host key: /etc/ssh/ssh_host_dsa_key Disabling protocol version 2. Could not load host key sshd: no hostkeys available -- exiting.'
Dec 3 09:47:23     php: : New alert found: pfSense has completed creating your SSH keys. SSH is now started.
Dec 3 09:47:07     sshlockout[63359]: sshlockout/webConfigurator v2.0 starting up
Dec 3 09:47:07     sshlockout[63359]: sshlockout/webConfigurator v2.0 starting up
Dec 3 09:47:07     login: login on console as root
Dec 3 09:47:02     php: : Resyncing configuration for all packages.
Dec 3 15:47:00     php: : Creating rrd update script
Dec 3 15:47:00     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt4-queuedrops.rrd -t :opt4:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:47:00     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt4-queues.rrd -t :opt4:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:59     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt3-queuedrops.rrd -t :opt3:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:59     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt3-queues.rrd -t :opt3:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:59     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt2-queuedrops.rrd -t :opt2:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:59     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt2-queues.rrd -t :opt2:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:58     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-queuedrops.rrd -t :opt1:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:58     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-queues.rrd -t :opt1:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:57     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-queuedrops.rrd -t :lan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:57     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-queues.rrd -t :lan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:56     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-queuedrops.rrd -t :wan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 15:46:56     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-queues.rrd -t :wan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 09:46:54     last message repeated 5 times
Dec 3 09:46:54     kernel: Bump sched buckets to 64 (was 0)
Dec 3 15:46:44     check_reload_status: reloading filter
Dec 3 09:46:34     apinger: ALARM: WANGW(209.32.69.81) *** down ***
Dec 3 09:46:31     kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding enabled, default to accept, logging disabled
Dec 3 09:46:30     dnsmasq[56810]: read /etc/hosts - 45 addresses
Dec 3 15:46:30     check_reload_status: updating all dyndns
Dec 3 09:46:29     dnsmasq[56810]: using nameserver 206.9.80.11#53
Dec 3 09:46:29     dnsmasq[56810]: using nameserver 206.9.80.12#53
Dec 3 09:46:29     dnsmasq[56810]: using nameserver 134.129.111.111#53
Dec 3 09:46:29     dnsmasq[56810]: using nameserver 134.129.201.29#53
Dec 3 09:46:29     dnsmasq[56810]: reading /etc/resolv.conf
Dec 3 09:46:29     dnsmasq[56810]: compile time options: no-IPv6 GNU-getopt no-DBus I18N DHCP TFTP
Dec 3 09:46:29     dnsmasq[56810]: started, version 2.55 cachesize 10000
Dec 3 09:46:29     dhcpleases: Could not deliver signal HUP to process because its pidfile does not exist, No such file or directory.
Dec 3 09:46:29     dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Dec 3 09:46:29     dhcpd: All rights reserved.
Dec 3 09:46:29     dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Dec 3 09:46:29     dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Dec 3 15:46:27     php: : ROUTING: add default route to 209.32.69.81
Dec 3 15:46:25     php: : The command '/usr/sbin/pw groupadd -g -M '' 2>&1' returned exit code '65', the output was 'pw: group name required'
Dec 3 09:46:24     apinger: Starting Alarm Pinger, apinger(35670)
Dec 3 09:46:24     last message repeated 5 times
Dec 3 09:46:24     kernel: Bump sched buckets to 64 (was 0)
Dec 3 09:46:23     kernel: load_dn_sched dn_sched PRIO loaded
Dec 3 09:46:23     kernel: load_dn_sched dn_sched WF2Q+ loaded
Dec 3 09:46:23     kernel: load_dn_sched dn_sched RR loaded
Dec 3 09:46:23     kernel: load_dn_sched dn_sched QFQ loaded
Dec 3 09:46:23     kernel: load_dn_sched dn_sched FIFO loaded
Dec 3 09:46:09     php: : New alert found: pfSense has started creating your SSH keys. SSH Startup will be delayed. Please note that reloading the filter rules and changes will be delayed until this operation is completed.

I added a little bit of debugging code to the end of the config_mount_ro function in config.lib.inc, send off an alert when the end of that function is reached, after the filesystem has been remounted ro.

It looks like the filesystem is getting remounted as ro by another process before the /etc/sshd script is done creating all the keys. Race condition?

• Some sort of lock file with reference counting for running the config_mount_rw and ro functions. This would solve any future similar race conditions.
• create they keys in /tmp first, then mount rw, copy files, then mount ro. This would at least make the window smaller. Would probably still need to call config_mount_rw right before the copy though.
• Delay the sshd regeneration until after all other startup scripts.
• break out the sshd regeneration into a seperate script that gets run after all other startup scripts so there is no way for it to conflict with another script.

• delete /conf/sshd
• delete /etc/ssh/ssh_host*
• reboot

Here are the logs:

Dec 3 13:48:44     php: : New alert found: pfSense mounted file system read only
Dec 3 13:48:44     php: : Reference 1000 is going negative, not doing unreference.
Dec 3 13:48:44     php: : New alert found: SSHD failed to start.
Dec 3 13:48:44     php: : The command '/usr/sbin/sshd' returned exit code '1', the output was 'Could not load host key: /etc/ssh/ssh_host_rsa_key Could not load host key: /etc/ssh/ssh_host_dsa_key Disabling protocol version 2. Could not load host key sshd: no hostkeys available -- exiting.'
Dec 3 13:48:44     php: : New alert found: pfSense has completed creating your SSH keys. SSH is now started.
Dec 3 13:48:30     php: /diag_logs.php: Successful webConfigurator login for user 'admin' host: 192.168.227.36
Dec 3 13:48:16     sshlockout[55467]: sshlockout/webConfigurator v2.0 starting up
Dec 3 13:48:16     sshlockout[55467]: sshlockout/webConfigurator v2.0 starting up
Dec 3 13:48:16     login: login on console as root
Dec 3 13:48:16     php: : New alert found: pfSense mounted file system read only  <---- :(
Dec 3 13:48:11     php: : Resyncing configuration for all packages.
Dec 3 13:48:10     php: : Creating rrd update script
Dec 3 13:48:09     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt4-queuedrops.rrd -t :opt4:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:09     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt4-queues.rrd -t :opt4:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:09     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt3-queuedrops.rrd -t :opt3:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:09     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt3-queues.rrd -t :opt3:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:08     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt2-queuedrops.rrd -t :opt2:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:08     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt2-queues.rrd -t :opt2:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:08     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-queuedrops.rrd -t :opt1:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:08     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-queues.rrd -t :opt1:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:07     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-queuedrops.rrd -t :lan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:07     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-queues.rrd -t :lan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:07     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-queuedrops.rrd -t :wan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:07     php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-queues.rrd -t :wan:qInternet:qACK:qOthersDefault:qP2P:qOthersHigh:qOthersLow N:U:U:U:U:U:U:U' returned exit code '1', the output was 'ERROR: unknown DS name '''
Dec 3 13:48:05     last message repeated 5 times
Dec 3 13:48:05     kernel: Bump sched buckets to 64 (was 0)
Dec 3 13:47:56     check_reload_status: reloading filter
Dec 3 13:47:46     apinger: ALARM: WANGW(209.32.69.81) *** down ***
Dec 3 13:47:42     kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding enabled, default to accept, logging disabled
Dec 3 13:47:41     check_reload_status: updating all dyndns
Dec 3 13:47:41     dnsmasq[58618]: read /etc/hosts - 46 addresses
Dec 3 13:47:41     dnsmasq[58618]: using nameserver 206.9.80.11#53
Dec 3 13:47:41     dnsmasq[58618]: using nameserver 206.9.80.12#53
Dec 3 13:47:41     dnsmasq[58618]: using nameserver 134.129.111.111#53
Dec 3 13:47:41     dnsmasq[58618]: using nameserver 134.129.201.29#53
Dec 3 13:47:41     dnsmasq[58618]: reading /etc/resolv.conf
Dec 3 13:47:41     dnsmasq[58618]: compile time options: no-IPv6 GNU-getopt no-DBus I18N DHCP TFTP
Dec 3 13:47:41     dnsmasq[58618]: started, version 2.55 cachesize 10000
Dec 3 13:47:41     dhcpleases: Could not deliver signal HUP to process because its pidfile does not exist, No such file or directory.
Dec 3 13:47:40     dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Dec 3 13:47:40     dhcpd: All rights reserved.
Dec 3 13:47:40     dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Dec 3 13:47:40     dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Dec 3 13:47:38     php: : ROUTING: add default route to 209.32.69.81
Dec 3 13:47:37     php: : The command '/usr/sbin/pw groupadd -g -M '' 2>&1' returned exit code '65', the output was 'pw: group name required'
Dec 3 13:47:36     apinger: Starting Alarm Pinger, apinger(37311)
Dec 3 13:47:36     last message repeated 5 times
Dec 3 13:47:35     kernel: Bump sched buckets to 64 (was 0)
Dec 3 13:47:35     kernel: load_dn_sched dn_sched PRIO loaded
Dec 3 13:47:35     kernel: load_dn_sched dn_sched WF2Q+ loaded
Dec 3 13:47:35     kernel: load_dn_sched dn_sched RR loaded
Dec 3 13:47:35     kernel: load_dn_sched dn_sched QFQ loaded
Dec 3 13:47:35     kernel: load_dn_sched dn_sched FIFO loaded
Dec 3 13:47:22     php: : New alert found: pfSense has started creating your SSH keys. SSH Startup will be delayed. Please note that reloading the filter rules and changes will be delayed until this operation is completed.
Dec 3 13:47:21     kernel: pflog0: promiscuous mode enabled
Dec 3 13:47:20     kernel: glxsb0: <AMD Geode LX Security Block (AES-128-CBC, RNG)> mem 0xefff4000-0xefff7fff irq 9 at device 1.2 on pci0
Dec 3 13:47:20     kernel: Trying to mount root from ufs:/dev/ufs/pfsense1

As far as I know, there is already reference counting for the conf_mount_rw/ro functions, so that it isn't mounted read-only until there has been enough conf_mount_ro calls. There must be something that has one too many calls to that function.

Right now it is left rw (not sure why it was done; this is only temporary, it will be fixed). When that is changed back, I think my latest commit for this should fix the issue described by Josh Stompro.

Check the latest image and see if this has been fixed.

I'm not sure how to test this since in the latest snapshot the / mount is still set to rw by default. Erik, you mentioned that it was being left rw 4 months ago, and that it was going to be fixed. Can you tell me where that is set? My fstab shows / being mounted ro, but that must get changed later in the boot cycle since when I look at the filesystem it is mounted rw.

Thanks
Josh

I now see bug #1279, which discusses the rw nanobsd mount issue.
Josh