pfSense

Note this was after adding a 3rd party SSD.

The clean installation procedure on the 3rd party SSD went smoothly, but the issue persisted with no other side effects.

Danilo Zrenjanin wrote in #note-3:

The clean installation procedure on the 3rd party SSD went smoothly, but the issue persisted with no other side effects.

What make and model is the SSD?

Installing the SSD drive in another port did not resolve the issue. It behaved in the same way.

The TAC ticket number for reference is:
2157407569

same issue here. 6100, 23.09.1

onboard mmc died, added a WD SN520 NVME SSD.

same console output

Another one TAC ticket with the same issue
2336421939

Another user having this issue on a 5100. 2396258146

Customer reported this issue utilizing a Dogfish 2242 M.2 SSD. Model SSDMCEAC060B3A. Customer ticket 2396258146.

Another customer having this issue on a 5100 with a SHARKSPEED SSD M.2 2242 128GB. 2423567446

Same issue. Factory Netgate 6100. No add-ons or changes. 23.09.1
Have NOT changed the internal storage. Factory everything.

Another user encountered the same issue with their Netgate 6100 device after installing the Transcend TS256GMTE460T NVMe SSD drive.

The internal ticket reference number for this issue is #2488837032.

The 256 GB WD NVMe SSD performs without any issues when used with the Netgate 6100.

#2694043009 A 4100 using a Transcend TS128GMTE452T, but doesn't show the same "can't unload" message. This instead looks normal but it never actually reboots.

Netgate pfSense Plus is rebooting now.
pflog0: promiscuous mode disabled
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 0 done
All buffers synced.
Uptime: 2m38s
uhub0: detached

#2694043009 The latest test shows Khelp module "ertt" can't unload until its refcount drops from 5 to 0.

Enter an option:
Message from syslogd@pfSense at Jun 28 07:01:32 ...
php-fpm[407]: /index.php: Successful login for user 'admin' from: 192.168.1.100 (Local Database)

Netgate pfSense Plus is now shutting down ...

net.inet.carp.allow: 0 -> 0
pflog0: promiscuous mode disabled
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 done
All buffers synced.
Uptime: 2m27s
Khelp module "ertt" can't unload until its refcount drops from 5 to 0.
ix3: link state changed to DOWN
ix3: link state changed to UP
uhub0: detached

Ran into the same issue today on a 6100.
The internal storage was no longer detected after a power outage, so we installed pfSense 24.03 to a USB stick as a temporary workaround.
The output looks the same as the other posts - the usb devices are detached and then nothing. It also needed to be manually rebooted when the the pfSense installation was finished.

We DO NOT get the "Khelp module "ertt" can't unload until its refcount drops from 5 to 0." message".

Same error as others above. 5100 with a TS32GMTS400S 32gb M.2, same as photo in original post.
This occured after using serial console to "update from console", updating v23.05.1 to 23.09.1

Danilo Zrenjanin wrote in #note-15:

The 256 GB WD NVMe SSD performs without any issues when used with the Netgate 6100.

Which specific model would that be? Quite odd that basically all SSDs show the issue but one doesn't...?

The issue persisted in 24.11.

Same here with a 5100 after change the original 8GB eMMC to a 128 GB SSD. It never reboots without removing the power completely....

Model Family: Silicon Motion based SSDs
Device Model: TS128GMTS430S

Same Problem here with 5100 and 24.11

facing the same issue too with a 4100. any workaround that can be used?

M O wrote in #note-8:

same issue here. 6100, 23.09.1

onboard mmc died, added a WD SN520 NVME SSD.

same console output

Chris W wrote in #note-16:

#2694043009 A 4100 using a Transcend TS128GMTE452T, but doesn't show the same "can't unload" message. This instead looks normal but it never actually reboots.

[...]

Andrew Almond wrote in #note-19:

Ran into the same issue today on a 6100.
The internal storage was no longer detected after a power outage, so we installed pfSense 24.03 to a USB stick as a temporary workaround.
The output looks the same as the other posts - the usb devices are detached and then nothing. It also needed to be manually rebooted when the the pfSense installation was finished.

We DO NOT get the "Khelp module "ertt" can't unload until its refcount drops from 5 to 0." message".

João Moreira wrote in #note-20:

same error here after change the original 16 GB eMMC to a 128 GB NVMe SSD

Wil M wrote in #note-26:

facing the same issue too with a 4100. any workaround that can be used?

possibly. can each of you open a ticket with TAC, referencing this thread and we'll take it from there?

Internal ticket related #38756047541

Internal ticket 45546069819 also experiencing this on 26.03.

Customer in ticket 45556712615 is also experiencing this issue with a TOSHIBA kbg30zms 256Gb NVME drive

Hi,

Same issue for me, Netgate 4100 EMMC is died.
I have install an TOSHIBA kbg30zms SSD.

When i reboot i got the following output from the console :

Netgate pfSense Plus is now shutting down ...
net.inet.carp.allow: 0 -> 0
pflog0: promiscuous mode disabled
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 done
All buffers synced.
Uptime: 13m27s
ix3: link state changed to DOWN
ix3: link state changed to UP
uhub0: detached

Then nothing and i have to manually power cycle the netgate 4100 for the reboot.

TAC Ticket number : 45556712615

EDIT : RESOLVED BY HARDWARE MODIFICATION
AFTER DESOLDERING THE EMMC THE FIREWALL WORK PERFECTLY FINE WITH A NVME DRIVE (reboot and instant console output at startup)
This modification isn't something everyone can do and requires equipment, but if you know someone who can do it, that solves the problem.

y B wrote in #note-33:

EDIT : RESOLVED BY HARDWARE MODIFICATION
AFTER DESOLDERING THE EMMC THE FIREWALL WORK PERFECTLY FINE WITH A NVME DRIVE (reboot and instant console output at startup)
This modification isn't something everyone can do and requires equipment, but if you know someone who can do it, that solves the problem.

I can confirm the same result on my Netgate 4100.

My unit had a Kingston eMMC chip and is now booting from an SK Hynix NVMe drive using a B+M adapter. Before removing the eMMC chip, the firewall would hang on reboot and require a manual power cycle. After desoldering the eMMC chip, rebooting from the CLI now works normally, with console output appearing immediately at startup.

This is obviously not a general user-friendly workaround and requires proper soldering equipment/experience, but in my case it resolved the reboot hang.

Jim Thompson wrote in #note-27:

Andrew Almond wrote in #note-19:

Ran into the same issue today on a 6100.
The internal storage was no longer detected after a power outage, so we installed pfSense 24.03 to a USB stick as a temporary workaround.
The output looks the same as the other posts - the usb devices are detached and then nothing. It also needed to be manually rebooted when the the pfSense installation was finished.

We DO NOT get the "Khelp module "ertt" can't unload until its refcount drops from 5 to 0." message".

João Moreira wrote in #note-20:

same error here after change the original 16 GB eMMC to a 128 GB NVMe SSD

Wil M wrote in #note-26:

facing the same issue too with a 4100. any workaround that can be used?

possibly. can each of you open a ticket with TAC, referencing this thread and we'll take it from there?

We have a 6100 that was reported with drive problems from a customer. Send them a NVMe to try to recover, they installed it and were able to install pfSense on the NVMe but afterwards had a strange error while booting the system, then after powering down the system won't even start again, it's now in the "orange-led-standby" state and won't start or do anything at all. No console output. Another power brick, removing the NVMe, or resetting via the side buttons, all tested, all nothing.

We also have a 4100 at another customer that seems to go the same route :/ As we read @jwt's comment in

https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far/318?_=1788504517658

was there ever a solution achieved with Silicom or something internal that we can get the hardware to boot up again at all?

Thanks!