Bug #15993
closed
Upgrade fails with NVMe drive installed
0%
Description
Hello fellow Redmine community members, I found a bug during upgrade to 25.11 the upgrade fails with my NVMe driver.
Loading kernel...
/boot/kernel/kernel text=0x318 text=0xcbe8a8 text=0x14683e8 data=0x19cff0 data=0x0+0x2ca000 0x8+0x19d088+0x8+0x1b8d4c
Loading configured modules...
/boot/kernel/zfs.ko text=0xce1b0 text=0x220bd8 data=0x3b0f0+0xaad44 0x8+0x49380+0x8+0x302ef
/boot/kernel/opensolaris.ko text=0x195e text=0xbf0 data=0x4d0+0x1a030 0x8+0xf78+0x8+0x87b
can't find '/etc/hostid'
can't find '/boot/entropy'
Using DTB provided by EFI at 0x8000000.
---<<BOOT>>---GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
WARNING: DTB version is 6.4 while kernel expects 6.8, please update the DTB in the ESP
Copyright (c) 1992-2024 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:12:13 UTC 2024
root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/aarch64/KRivbaXa/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBSD-src-plus-RELENG_24_11/arm64.aarch64/sys/pfSense arm64
FreeBSD clang version 18.1.6 (https://github.com/llvm/llvm-project.git llvmorg-18.1.6-0-g1118c2e05e67)
VT: init without driver.
module scmi already present!
real memory = 3602862080 (3435 MB)
avail memory = 3461632000 (3301 MB)
Starting CPU 1 (1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
wlan: mac acl policy registered
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
simplebus1: <Flattened device tree simple bus> on simplebus0
simple_mfd0: <Simple MFD (Multi-Functions Device)> mem 0x13800-0x138ff,0x13c00-0x13c1f on simplebus1
simple_mfd1: <Simple MFD (Multi-Functions Device)> mem 0x18800-0x188ff,0x18c00-0x18c1f on simplebus1
psci0: <ARM Power State Co-ordination Interface Driver> on ofwbus0
netgate0: <Netgate 2100>
netgate0: version: 0.1
gic0: <ARM Generic Interrupt Controller v3.0> mem 0x1d00000-0x1d0ffff,0x1d40000-0x1d7ffff,0x1d80000-0x1d81fff,0x1d90000-0x1d91fff,0x1da0000-0x1dbffff irq 27 on simplebus1
gpio0: <Armada 37x0 North Bridge GPIO Controller> mem 0x13800-0x138ff,0x13c00-0x13c1f irq 28,29,30,31,32,33,34,35,36,37,38,39 on simple_mfd0
gpiobus0: <OFW GPIO bus> on gpio0
gpio1: <Armada 37x0 South Bridge GPIO Controller> mem 0x18800-0x188ff,0x18c00-0x18c1f irq 40,41,42,43,44 on simple_mfd1
gpiobus1: <OFW GPIO bus> on gpio1
gpiomdio0: <GPIO MDIO bit-banging Bus driver> at pins 21,23 on gpiobus1
gpiomdio0: MDC pin: 21, MDIO pin: 23
a37x0-xtal0: <Marvell Armada 3700 Oscillator> mem 0x13800-0x138ff,0x13c00-0x13c1f on simple_mfd0
generic_timer0: <ARMv8 Generic Timer> irq 0,1,2,3 on ofwbus0
Timecounter "ARM MPCore Timecounter" frequency 12500000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 12500000 Hz quality 1000
a37x0_tbg0: <Marvell Armada 3700 time base generators> mem 0x13200-0x132ff on simplebus1
a37x0_nb_periph_driver0: <marvell,armada-3700-nb-periph-clock> mem 0x13000-0x130ff on simplebus1
a37x0_sb_periph_driver0: <marvell,armada-3700-sb-periph-clock> mem 0x18000-0x180ff on simplebus1
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
pmu0: <Performance Monitoring Unit> irq 4 on ofwbus0
spi0: <Armada 37x0 SPI controller> mem 0x10600-0x10fff irq 6 on simplebus1
iichb0: <Marvell Armada 37x0 IIC controller> mem 0x11000-0x11023 irq 7 on simplebus1
iicbus0: <OFW I2C bus> on iichb0
iic0: <I2C generic I/O> on iicbus0
iichb1: <Marvell Armada 37x0 IIC controller> mem 0x11080-0x110a3 irq 8 on simplebus1
iicbus1: <OFW I2C bus> on iichb1
iic1: <I2C generic I/O> on iicbus1
nxprtc0: <NXP PCF8563 RTC> at addr 0xa2 on iicbus1
gpio2: <ISSI IS31FL3199 9 channel light effect LED driver> at addr 0xce on iicbus1
gpiobus2: <OFW GPIO bus> on gpio2
gpioc2: <GPIO controller> on gpio2
uart0: <Marvell Armada 3700 UART> mem 0x12000-0x121ff irq 9,10,11 on simplebus1
uart0: console (115200,n,8,1)
gpioc0: <GPIO controller> on gpio0
gpioc1: <GPIO controller> on gpio1
mvneta0: <NETA controller> mem 0x30000-0x33fff irq 14 on simplebus1
mvneta0: version is 10
miibus0: <MII bus> on gpiomdio0
mv88e151x0: <Marvell 88E1512 Gigabit PHY> PHY 1 on miibus0
mv88e151x0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseSX, 1000baseSX-FDX, auto
mvneta0: Ethernet address: 90:ec:77:0d:da:25
mvneta1: <NETA controller> mem 0x40000-0x43fff irq 15 on simplebus1
mvneta1: version is 10
mdio0: <MDIO> on mvneta1
mdioproxy0: <MII/MDIO proxy, MDIO side> on mdio0
e6000sw0: <Marvell 88E6141> on mdio0
e6000sw0: single-chip addressing mode
e6000sw0: PHY at port 1
miibus1: <MII bus> on e6000sw0
e1000phy0: <Marvell 88E1000 Gigabit PHY> PHY 17 on miibus1
e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: PHY at port 2
miibus2: <MII bus> on e6000sw0
e1000phy1: <Marvell 88E1000 Gigabit PHY> PHY 18 on miibus2
e1000phy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: PHY at port 3
miibus3: <MII bus> on e6000sw0
e1000phy2: <Marvell 88E1000 Gigabit PHY> PHY 19 on miibus3
e1000phy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: PHY at port 4
miibus4: <MII bus> on e6000sw0
e1000phy3: <Marvell 88E1000 Gigabit PHY> PHY 20 on miibus4
e1000phy3: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: CPU port at 5
e6000sw0: fixed port at 5
e6000sw0: switch is ready.
etherswitch0: <Switch controller> on e6000sw0
mvneta1: Ethernet address: 90:ec:77:0d:da:26
xhci0: <Generic USB 3.0 controller> mem 0x58000-0x5bfff irq 16 on simplebus1
xhci0: 32 bytes context size, 32-bit DMA
usbus0 on xhci0
ehci0: <Marvell Integrated USB 2.0 controller> mem 0x5e000-0x5ffff irq 17 on simplebus1
usbus1: EHCI version 1.0
usbus1 on ehci0
sdhci_xenon0: <Armada Xenon SDHCI controller> mem 0xd8000-0xd82ff,0x17808-0x1780b irq 25 on simplebus1
mmc0: <MMC/SD bus> on sdhci_xenon0
ahci0: <AHCI SATA controller> mem 0xe0000-0xe1fff irq 26 on simplebus1
ahci0: AHCI v1.30 with 1 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
pcib0: <Marvell Armada 3700 PCIe Bus Controller> mem 0xd0070000-0xd008ffff irq 5 on simplebus0
pcib0: link up
pci0: <OFW PCI bus> on pcib0
nvme0: <Generic NVMe Device> mem 0xe8000000-0xe8003fff at device 0.0 on pci0
Fatal data abort:
x0: 0x0000000000000000
x1: 0x00000000e8000000
x2: 0x0000000000000000
x3: 0x0000000000000000
x4: 0x0000000000020000
x5: 0x0000000000000006
x6: 0x0000000000000002
x7: 0xffff0000022c9520 (initstack + 0x3520)
x8: 0xffff000002221b00 (memmap_bus + 0x0)
x9: 0xffff000000b6a350 (generic_bs_r_4 + 0x0)
x10: 0xffff00000219e3b8 (bus_get_domain_desc + 0x0)
x11: 0xffff00000219e3b8 (bus_get_domain_desc + 0x0)
x12: 0xffff0000021b1ba8 (lock_class_mtx_spin + 0x0)
x13: 0xfefefefefeff0f00
x14: 0x0000000000000000
x15: 0x0000000000000000
x16: 0xffff0000006bc734 (bus_generic_get_domain + 0x0)
x17: 0x0000000000000000
x18: 0xffff0000022c95a0 (initstack + 0x35a0)
x19: 0xffff00009cc50000 ($d.6 + 0x99ebe5b8)
x20: 0xffffa00001c83200
x21: 0xffff00009cc50048 ($d.6 + 0x99ebe600)
x22: 0xffff00009cc69000 ($d.6 + 0x99ed75b8)
x23: 0xffff00009cc50068 ($d.6 + 0x99ebe620)
x24: 0xffffa00001c83300
x25: 0xffff000000cf4c5b (notify.prefix + 0x326f7)
x26: 0xffffa000019478a0
x27: 0xffffa000015c7620
x28: 0x0000000000000002
x29: 0xffff0000022c9620 (initstack + 0x3620)
sp: 0xffff0000022c95a0
lr: 0xffff0000002f7e38 (nvme_ctrlr_construct + 0x70)
elr: 0xffff000000b6a358 (generic_bs_r_4 + 0x8)
spsr: 0x00000000600000c5
far: 0x00000000e8000000
esr: 0x0000000096000004
panic: vm_fault failed: 0xffff000000b6a358 error 1
cpuid = 0
time = 1
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at kdb_enter+0x48: str xzr, [x19, #2944]
db>
Can we add a script that auto disables the driver until after the install is complete ? I use this drive over movie adapter to NVMe m.2
set hint.nvme.0.disabled=1
boot
Ref:
https://forum.netgate.com/topic/195988/2100-max-upgrade-24-03-to-24-11-failed
Files
Updated by Jim Pingle 3 months ago
- Status changed from New to Rejected
There are systems which require NVMe to function as that is their only drive, so disabling it automatically at any point is not an option.
Per the linked thread, they are attempting to use an NVMe drive on a Netgate 2100. The Netgate 2100 does not support NVMe. If it worked in the past, it was by chance.
See the notes on https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/m-2-sata-installation.html for starters.
Updated by Jonathan Lee 3 months ago
@Jim Pringle
I am using a mpcie to nvme adapter on the mpcie slot that was used for wifi. It does work perfectly however during the upgrade it is failing. So, it does support it if and only if you utilize an adapter again my main software is not running on it. The OS is using the SSD M.2 B built in slot for the pfsense software. I am using the NVMe drive with logs, similar to adding a second disk on a PC however something in the update software is failing during the update with that drive installed, I am using an Intel Optane H10 in that slot, it works perfectly otherwise, I have linker files directing snort to use that as a log location as well as syslog-ng and squid cache and logs also. It does all logging to it. If the main update is going to the M.2 drive what would cause my system to fail on the driver? Is that a pfSense thing that if it sees that drive, it assumes it is a different device? Even if it is not supported would you not agree that if a system has one it should not fail? I mean other OS systems if you add a second disk it will never fail on updates. What will cause this error? Yes, I agree set hint.nvme.0.disabled=1 would cause issues with upper class systems that use NVMe for everything I did not think about that. Is there a way for users that have a second drive to temp disable it and or ignore the drive until software is updated the firmware is device specific when we call into TAC, so in theory a simple code change could fix this. I was told I am the only one that has a 2100 with a NVMe drive on the forum and even at Netgate offices so I am willing to try anything. It does work, and it is not by chance as it took a lot of trial and error to get it online with different partitions mount commands etc. I even have added add a fan to make it cool down.
Updated by Jim Pingle 3 months ago
That you managed to make it function partially does not indicate it is "supported". It may have appeared functional, but Netgate does not support that configuration, and it's explicitly documented as not supported.
There are any number of things which the hardware and operating system may be capable of doing to varying degrees but that alone does not mean they are intended to work or indicate that they are things for which Netgate can offer assistance.
Updated by Jonathan Lee 3 months ago
It is the everything bagel 2100-MAX, we can call it the SG-2100-ULTRA edition .. comes with heat sinks a nvme drive, an adapter and a small fan, that was the hard part getting the fan to work. (fan not included in photo)