pfSense

Those additional backtraces in comment #1 look totally different, and there's no indication that these are the same issue.

The address suggests we're crashing on `ifp = r->rpool.cur->kif ? r->rpool.cur->kif->pfik_ifp : NULL;` in pf_route(), the we dereference the r->pool.cur->kif pointer, because it's not NULL, but 0x12. That's very, very odd, in that we ought to have a valid rule pointer (and almost certainly would have crashed earlier if we didn't), but somehow we also seem to have an invalid kpool.

I'd be very interesting to be able to reproduce this or at least get a full kernel core dump to investigate.

The affected user has very helpfully provided a core dump, which shows a couple of things.
Firstly it confirms what I gathered from the minidump: we have a rule with a NULL rpool.cur. This rule has rule number -1, which indicates that it's the V_pf_default_rule.

That happens when the rules on the pfsync hosts are different (see pfsync_state_import()). It looks like V_pf_default_rule doesn't get its rpool correctly initialised, which in turn causes this panic.
That's a fairly straightforward fix, which I'm working on, as well as a test case to reproduce this crash (and to ensure we don't regress on this).

There are no more crashes on the latest snapshots. Many thanks to all participants.

Does this patch apply to the current beta builds? I have tried one beta build from 13 Oct and it crashes the same way.

Yes, the relevant patch is in the 23.09 branch. What version are you running and what is the full backtrace you're getting?

db:1:pfs> bt
Tracing pid 12 tid 100062 td 0xfffffe00c641f560
kdb_enter() at kdb_enter+0x32/frame 0xfffffe001b1e2600
vpanic() at vpanic+0x163/frame 0xfffffe001b1e2730
panic() at panic+0x43/frame 0xfffffe001b1e2790
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe001b1e27f0
trap_pfault() at trap_pfault+0xae/frame 0xfffffe001b1e2860
calltrap() at calltrap+0x8/frame 0xfffffe001b1e2860
--- trap 0xc, rip = 0xffffffff80fd28d8, rsp = 0xfffffe001b1e2930, rbp = 0xfffffe001b1e29e0 ---
pf_route() at pf_route+0x768/frame 0xfffffe001b1e29e0
pf_test() at pf_test+0x1014/frame 0xfffffe001b1e2b80
pf_check_out() at pf_check_out+0x22/frame 0xfffffe001b1e2ba0
pfil_mbuf_out() at pfil_mbuf_out+0x58/frame 0xfffffe001b1e2bd0
ip_output() at ip_output+0xce6/frame 0xfffffe001b1e2cd0
ip_forward() at ip_forward+0x413/frame 0xfffffe001b1e2d80
ip_input() at ip_input+0x814/frame 0xfffffe001b1e2de0
swi_net() at swi_net+0x19b/frame 0xfffffe001b1e2e60
ithread_loop() at ithread_loop+0x266/frame 0xfffffe001b1e2ef0
fork_exit() at fork_exit+0x82/frame 0xfffffe001b1e2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001b1e2f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:1:pfs> show registers
cs 0x20
ds 0x3b
es 0x3b
fs 0x13
gs 0x1b
ss 0x28
rax 0x12
rcx 0xffffffff814b1a68
rdx 0xffffffff814a2042
rbx 0x100
rsp 0xfffffe001b1e2600
rbp 0xfffffe001b1e2600
rsi 0x80
rdi 0xffffffff83024090 cnputs_mtx
r8 0
r9 0
r10 0
r11 0
r12 0
r13 0
r14 0xffffffff81416f35
r15 0xfffffe00c641f560
rip 0xffffffff80d33612 kdb_enter+0x32
rflags 0x86
kdb_enter+0x32: movq $0,0x234c273(%rip)
db:1:pfs> show pcpu
cpuid = 5
dynamic pcpu = 0xfffffe009ca4fcc0
curthread = 0xfffffe00c641f560: pid 12 tid 100062 critnest 1 "swi1: netisr 5"
curpcb = 0xfffffe00c641fa80
fpcurthread = none
idlethread = 0xfffffe00c6369000: tid 100008 "idle: cpu5"
self = 0xffffffff84215000
curpmap = 0xffffffff83023ab0
tssp = 0xffffffff84215384
rsp0 = 0xfffffe001b1e3000
kcr3 = 0x80000000c5c5c001
ucr3 = 0xffffffffffffffff
scr3 = 0x23472bb5d
gs32p = 0xffffffff84215404
ldt = 0xffffffff84215444
tss = 0xffffffff84215434
curvnet = 0xfffff80001177f40
spin locks held:

Yes, but what version are you running?
Post the output of "uname -a" and "pkg info pfSense-kernel-pfSense".

Sorry just went out of my head…

FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09-n256151-106588946ac: Mon Oct 16 03:09:09 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/amd64/nZsGV1oA/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/sources/FreeBSD-src-plus-RELENG_23_09/amd64.amd64/sys/pfSense-DEBUG amd64

pfSense-kernel-pfSense-23.09.b.20231016.0231
Name : pfSense-kernel-pfSense
Version : 23.09.b.20231016.0231
Installed on : Mon Oct 16 14:34:32 2023 EEST
Origin : security/pfSense-kernel
Architecture : FreeBSD:14:amd64
Prefix : /
Categories : security
Licenses : APACHE20
Maintainer : development@pfsense.org
WWW : http://www.pfsense.org/
Comment : pfSense kernel (pfSense)
Annotations :
repo_type : binary
repository : pfSense-core
Flat size : 87.4MiB
Description :
pfSense kernel (pfSense)

WWW: http://www.pfsense.org/

Cheers, that helped!

I think I see what happened here. Basically I fixed the problem upstream and missed a case in the pfsense tree. At least it ought to be a trivial fix.

Yes, looks like it is ok now. No more crashes on beta 23.09