Todo #4847

NanoBSD Image Flash Block Misalignment

Added by ky41083 - over 5 years ago. Updated over 3 years ago.

Operating System
Target version:
Start date:
Due date:
% Done:


Estimated time:


pfSense NanoBSD images are not flash block aligned. This causes significant slowdown during extended write disk activity (i.e. upgrade installs, package installs, NanoBSD slice duplication) on non-SSD flash based media. Research shows reads can also be negatively effected, but to a much lesser degree.

pfSense installable is also effected, but is not my focus as the fix will be much different. The installer picks the alignment, and is not meant to be run on flash media anyhow. With NanoBSD, the builder picks the alignment.

Lengthy discussion here:


#1 Updated by ky41083 - over 5 years ago

The change for NanoBSD would be implemented in the build system. The fdisk command that creates the initial MBR partition would have to be told to start that partition on sector 64, rather than sector 63 (default) as it is now. Also, the gap between slices needs to be 64, rather than 63, sectors to keep the second slice and config slice aligned the same.

Given the builder currently uses fdisk... The MBR start, and gap between slices (actually the entire disk layout), can be taken care of with a simple config file using "fdisk -f configfile". Then future changes can simply be made to said config file.

This will work as long as the fake device used to build NanoBSD from uses 512-byte sector emulation. I'm assuming NanoBSD is built either in a mounted image, or in a RAM disk which is then dumped to an image. FreeBSD supports setting sector emulation size for both of these devices, if necessary. For example, when the build server eventually uses true 4K drives, this will have to be set to 512-byte sectors in order to not throw off the sector math.

#2 Updated by ky41083 - over 5 years ago

The upgrade scenario for NanoBSD...

In the research I've done, as far as moving the entire MBR partition down by one sector, along with everything in it, even in blind sector copy mode, isn't supported by FreeBSD, or even the likes of Gparted Live. This would not be an acceptable upgrade solution anyhow, but shows how impossible modifying existing installs currently is.

Everything I found basically said the "solution" is to build a new disk and copy the contents of the old one to it. Again, this change should not even be attempted during an upgrade. Upgrades will simply retain the old disk alignment indefinitely, which shouldn't require any changes since upgrades are done per slice, without consideration for disk sector layout. This is no worse than the current situation.

The only bug free solution to getting the new, proper alignment, will be a config backup, full NanoBSD image reflash, and config restore. I know, not ideal, but it really appears to be the only full proof error free way of doing this. At least future installs will get correct alignment, which is infinitely better than the current situation. Again, upgrades should be unaffected here also, as they are done per slice.

#3 Updated by ky41083 - over 5 years ago

Keith Hough wrote:

start that partition on sector 64, rather than sector 63 (default) as it is now.

Got ahead of myself here. Forgot to mention that Linux, Windows, ESXi, and almost every other current OS, choose to start the very first partition at sector 2048, or 1MB, for a sort of "future proofing" of alignment, as flash devices use varying erase sector sizes, and 1MB aligns to all of them. Sector 64 only aligns to disks using 4K or smaller sectors. If we are going to make this change, I would think we want it to be as compatible / future proof as possible.

#4 Updated by Jim Thompson over 5 years ago

only problem here (assuming it works, and is useful) is that setting to sector 2048 probably renders a lot of old hardware unbootable.

#5 Updated by ky41083 - over 5 years ago

The boot code and MBR partition tables would remain where they are, in sector 0 / 1. If a system was going to have issues booting from a partition starting at sector 2048, such a system would also be completely unable to boot from NanoBSD slice 2.

Are there any systems you know of that can boot from NanoBSD slice 1, but fail to boot from slice 2?

#6 Updated by Jim Pingle over 5 years ago

Keith Hough wrote:

Are there any systems you know of that can boot from NanoBSD slice 1, but fail to boot from slice 2?

Nothing I can think of that should, by most any standard, still be in service. #1 example is the PC Engines WRAP. At this point it would fail to run a current version for other reasons (lack of RAM, for example) so it shouldn't be considered something to preserve. We even removed the directions that some had used to run NanoBSD on the wrap from one slice only (no upgrades possible)

I'd love to hear about other examples if anyone knows of any.

#7 Updated by ky41083 - over 5 years ago

I completely agree.

I would also love to hear about any examples of systems that can currently run pfSense 2.2, but cannot boot from slice 2 due to a hardware limitation.

#8 Updated by ky41083 - over 5 years ago

Want to add while I'm here, in case some don't read the linked thread. Per the very first reference listed, the beginning of the first MBR partition should optimally start at sector 8192, or 4MB, to best optimize for SD cards as well. pfSense NanoBSD target platform is moving more and more toward SD storage, especially with the APU platform now replacing ALIX. Since this does not un-optimize for CF, or any other type of flash disk (it might even help, depending on the controller), it would be the ideal solution.

The gap between partitions would use 64 sectors, or 32K, since 32K will align just fine with about any flash erase block size. And should not change anyways unless we are going to increase the UFS fragment size from 4K.

Since the UFS filesystem already uses 4K fragments, 4K aligns with flash erase block sizes 4K and larger, 4K is the most common minimum flash erase block size (today), and anything larger I think would waste unacceptable amounts of disk space, I think this should not change.

#9 Updated by ky41083 - over 5 years ago

Fun Fact. This is the FACTORY format of a SanDisk microSDXC 64GB UHS-1 card I just got as a warranty replacement. The original physically identical card that failed reported an optimal erase block size of 8MB, and I have no idea what offset the factory partition used (never checked).

The new replacement card is reporting an optimal erase block size of 64MB, and the following is the absolutely untouched factory partition layout from both Windows and Linux...

[Linux fdisk -l]
Disk /dev/sdd: 63.9 GB, 63864569856 bytes
255 heads, 63 sectors/track, 7764 cylinders, total 124735488 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sdd1 32768 124735487 62351360 7 HPFS/NTFS/exFAT

[Windows 8.1 Diskpart]
DISKPART> det disk

Disk ID: 00000000
Type : USB
Status : Online
Path : 0
Target : 0
LUN ID : 1
Location Path : UNAVAILABLE
Current Read-only State : No
Read-only : No
Boot Disk : No
Pagefile Disk : No
Hibernation File Disk : No
Crashdump Disk : No
Clustered Disk : No

Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 5 G exFAT Removable 59 GB Healthy

DISKPART> list part

Partition ###  Type              Size     Offset
------------- ---------------- ------- -------
Partition 1 Primary 59 GB 16 MB

C:\Windows\system32>chkdsk g:
The type of the file system is exFAT.
Volume Serial Number is 6564-3231
Windows is verifying files and folders...
File and folder verification is complete.

Windows has scanned the file system and found no problems.
No further action is required.

62334976 KB total disk space.
128 KB in 1 files.
256 KB in 2 indexes.
0 KB in bad sectors.
256 KB in use by the system.
62334336 KB available on disk.
131072 bytes in each allocation unit.
486992 total allocation units on disk.
486987 allocation units available on disk.

It seems flash controllers are being optimized very differently over time, even within the exact same product. Just adding info, the more understood the better.

#10 Updated by ky41083 - over 5 years ago

Any chance 2.3 will bring a NanoBSD installer? It would be so much easier if alignment / fragment size could be chosen manually.

#11 Updated by ky41083 - over 5 years ago

About time. Upstream:

Support for detecting and implementing aligning partitions on 1Mb boundaries has been added to bsdinstall(8). [r285721]

#12 Updated by ky41083 - over 5 years ago

Finally got around to setting up a current FreeBSD VM. Here is your example command for aligning MBR partitions to 4K:

gpart add -t freebsd-ufs -a4k -s20g ada0s1


Issue confirmed, solution provided.

#13 Updated by Chris Buechler about 5 years ago

  • Category set to Operating System

#14 Updated by drbob - almost 5 years ago

ky41083 - wrote:

gpart add -t freebsd-ufs -a4k -s20g ada0s1

Note that for mbr partitions kern.geom.part.mbr.enforce_chs=0 must be set otherwise gpart follows the MBR spec and adjusts the alignment to a CHS boundary.


#15 Updated by Renato Botelho almost 5 years ago

  • Assignee set to Renato Botelho

#16 Updated by Renato Botelho over 3 years ago

  • Status changed from New to Closed

nanobsd is dead in 2.4 and future versions

Also available in: Atom PDF