pfSense

Mounting read-only after mounting read-write can be very slow on recent NanoBSD images on 2.1, based on FreeBSD 8.3

Does not seem to affect all CF cards, but most, especially slower ones.

[2.1-DEVELOPMENT][root@pfSense.localdomain]/root(1): time /etc/rc.conf_mount_rw
0.457u 0.077s 0:01.04 50.0% 3184+477k 0+27io 0pf+0w
[2.1-DEVELOPMENT][root@pfSense.localdomain]/root(2): time /etc/rc.conf_mount_ro
0.441u 0.112s 0:37.01 1.4% 3101+450k 0+597io 0pf+0w

[2.1-DEVELOPMENT][root@pfSense.localdomain]/root(4): time mount -o rw /
0.005u 0.000s 0:00.00 0.0% 0+0k 0+3io 0pf+0w
[2.1-DEVELOPMENT][root@pfSense.localdomain]/root(5): time mount -o ro /
0.000u 0.021s 0:35.56 0.0% 160+704k 0+542io 0pf+0w

[2.1-DEVELOPMENT][root@pfSense.localdomain]/root(2): time /sbin/mount -u -w -o sync,noatime /
0.000u 0.006s 0:00.01 0.0% 0+0k 0+5io 0pf+0w
[2.1-DEVELOPMENT][root@pfSense.localdomain]/root(3): time /sbin/mount -u -r -f -o sync,noatime /
0.000u 0.018s 0:28.37 0.0% 64+2680k 0+407io 0pf+0w

Normally the read-only mount step should only take a few seconds, 5 or so.

As a consequence, many operations on the firewall take a very long time to work as it will pause for >30 seconds every time something re-mounts ro during the process. Reboot takes a very long time as well.

There is a lot more data in this forum thread:
http://forum.pfsense.org/index.php/topic,48256.msg257717.html#msg257717

A few more data points:

• Enabling TRIM on the slice - no effect, still slow
• Enabling softupdates on the slice - no effect, still slow
• Running sync;sync;sync; before mounting ro - no effect, still slow
• The mount process sticks in biowr state, which is disk i/o
• While mount is stuck, the system is somewhat unresponsive - can't run a process or connect with ssh, but does appear to continue routing.
• If you mount ro twice in a row, the second is fast, so it's only a problem when transitioning from read-write to read-only.

Truss on the slow mount process:
http://pastiebin.com/?page=p&id=4f9ace216b5cb

top -SH while mount is happening:
http://pastiebin.com/?page=p&id=4f9ae2a9bc080

systat -iostat :
http://pastiebin.com/?page=p&id=4f9ae3017b4ef

ststat -vmstat during and after :
http://pastiebin.com/?page=p&id=4f9ae3f4c746a

Because this is NanoBSD, DMA is already disabled and so is write caching.