pfSense

https://gitlab.netgate.com/pfSense/pfSense/-/merge_requests/1253

Very much looking forward to testing this! Will this make it into the 25.11 dev snapshots?

Yup, it's already in the 25.11 snapshots internally. :)

Tried to test this with System Patches on 25.11.b.20251028.1838 but I failed. Comments at https://github.com/pfsense/pfsense/commit/ae72f1ad8850ea48f07ee2788c2f1f4973d69667#commitcomment-169769189

(also noting the stray " char)

Since 25.11.b.20251111.2016 dropped today, I figured I'd try this again. Sadly, got an immediate fatal crash / pagefault again, after which my 6100 remained stuck in an eternal boot loop. I had to hook up the serial console, boot single user mode and edit the config.xml to remove the <eimnat></eimnat> from the outbound NAT rule.

Really not sure what's causing this. I took a 60fps screen recording of the serial console during this boot/crash in case that's useful. I also have the info.X (0-5) and textdump.tar.X files saved if those are useful—just let me know where to send them...

I put up some screenshots etc at this gist: https://gist.github.com/luckman212/a122228082bd14c37451ea1af5eb45dc

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 0c
fault virtual address   = 0x0
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff8102a81a
stack pointer           = 0x28:0xfffffe008c243570
frame pointer           = 0x28:0xfffffe008c243590
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (if_io_tqg_1)
rdi: fffff8026301488c rsi: 0000000094550512 rdx: 0000000094550517
rcx: 0000000000000000 r8:  00000000ef99b857  r9: 0000000060208000
rax: 0000000000000000 rbx: fffff80101814000 rbp: fffffe008c243590
r10: 0000000000000000 r11: 0000000000000040 r12: 000000000000e659
r13: fffffe008c2439e6 r14: fffff80263014888 r15: fffffe0092a7d958
trap number             = 12
panic: page fault
cpuid = 1
time = 1762962450
KDB: enter: panic
[ thread pid 0 tid 100012 ]
Stopped at      kdb_enter+0x33: movq    $0,0x1b3e052(%rip)

You can upload the crash dump files here:
https://nc.netgate.com/nextcloud/s/Dq9WxFFkCp4QiPN

Thanks - I just uploaded them. Also want to add, if any extra info is needed that can't be gleaned from the dumps and other info here and on the gist, I am happy to give remote access to the unit here to Netgate if requested.

I managed to reproduce the problem and have a fix pending review: https://reviews.freebsd.org/D53737

Kristof Provost That's excellent! I suscribed to that phabricator ID and will stay tuned for any way to test. Even after reading your description, I'm left curious what was unique about my setup that triggered this bug when seemingly nobody else did. Perhaps having the Tailscale pkg installed?

At least some NAT translations must have failed. This may be a configuration issue, or perhaps there's just so much traffic that you ran out of ports or perhaps there's another reason. I can't tell from the backtrace, and probably wouldn't even be able to tell from your full configuration file. I expect I'd have to spend a fair amount of time with both dtrace and network captures to figure that out.

Tested on pfSense-25.11.r.20251118.1708 and I have EIMNAT enabled now for outbound NAT.

As long as Static Port isn't enabled, it doesn't crash, however it seems that enabling both EIM and Static Port still leads to a panic, at least for some: https://forum.netgate.com/post/1230556

While I'm able to create the EIMNAT rule without kernel panicking now, not sure this feature is working for me. Tests are reporting "Port Restricted Cone NAT" or similar.

see https://forum.netgate.com/post/1230529 for more detail/screenshots.

Thanks Jim Pingle
In the other thread, Marcos was asking for me to test with the debug kernel - is that still useful at all or not anymore?

→ luckman212 wrote in #note-19:

Thanks Jim Pingle
In the other thread, Marcos was asking for me to test with the debug kernel - is that still useful at all or not anymore?

It may add some more useful data points but be aware that without the debug kernel I could only crash it on demand when killing states, but with the debug kernel it crashed when trying to pass any traffic. Static port had to be set on an EI rule in either case, though. If you do boot a debug kernel, only do so temporarily for the next boot, not persistently.

Just uploaded a crash dump to the nextcloud drop from the RC debug kernel if it helps

We have a patch pending that stops the crash, my test system has been stable overnight and all day today and I haven't been able to induce a crash. So at the moment there is no need for testing until that change is in a new RC build.