Project

General

Profile

Actions

Bug #15413

closed

Kernel panic in HA nodes when under high load

Added by Steve Wheeler 8 months ago. Updated 29 days ago.

Status:
Resolved
Priority:
Normal
Category:
FreeBSD
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
24.11
Release Notes:
Default
Affected Version:
2.7.2
Affected Architecture:
All

Description

Two 1541s running 23.09.1 in this example:

db:1:pfs> bt
Tracing pid 12 tid 100124 td 0xfffffe00e24d1560
kdb_enter() at kdb_enter+0x32/frame 0xfffffe01066f8820
vpanic() at vpanic+0x163/frame 0xfffffe01066f8950
panic() at panic+0x43/frame 0xfffffe01066f89b0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe01066f8a10
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe01066f8a70
calltrap() at calltrap+0x8/frame 0xfffffe01066f8a70
--- trap 0xc, rip = 0xffffffff80fb5e29, rsp = 0xfffffe01066f8b40, rbp = 0xfffffe01066f8ba0 ---
pf_test_state_udp() at pf_test_state_udp+0x2a9/frame 0xfffffe01066f8ba0
pf_test() at pf_test+0x110a/frame 0xfffffe01066f8d40
pf_check_in() at pf_check_in+0x27/frame 0xfffffe01066f8d60
pfil_mbuf_in() at pfil_mbuf_in+0x38/frame 0xfffffe01066f8d90
ip_input() at ip_input+0x3ae/frame 0xfffffe01066f8df0
swi_net() at swi_net+0x128/frame 0xfffffe01066f8e60
ithread_loop() at ithread_loop+0x257/frame 0xfffffe01066f8ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe01066f8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01066f8f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:1:pfs>  show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx         0xffffffff814591d9
rdx         0xffffffff844195ff
rbx                      0x100
rsp         0xfffffe01066f8820
rbp         0xfffffe01066f8820
rsi         0xfffffe01066f8290
rdi         0xffffffff82d40298  vt_conswindow+0x10
r8                        0x10
r9                        0x10
r10                        0xf
r11                       0x10
r12                          0
r13                          0
r14         0xffffffff813dc4fe
r15         0xfffffe00e24d1560
rip         0xffffffff80d38d62  kdb_enter+0x32
rflags                    0x82
kdb_enter+0x32: movq    $0,0x2344aa3(%rip)
db:1:pfs>  show pcpu
cpuid        = 8
dynamic pcpu = 0xfffffe00b4a6af00
curthread    = 0xfffffe00e24d1560: pid 12 tid 100124 critnest 1 "swi1: netisr 11" 
curpcb       = 0xfffffe00e24d1a80
fpcurthread  = none
idlethread   = 0xfffffe00e23fce40: tid 100011 "idle: cpu8" 
self         = 0xffffffff84018000
curpmap      = 0xffffffff83021ab0
tssp         = 0xffffffff84018384
rsp0         = 0xfffffe01066f9000
kcr3         = 0x8000000003f6f002
ucr3         = 0xffffffffffffffff
scr3         = 0x69eb1e858
gs32p        = 0xffffffff84018404
ldt          = 0xffffffff84018444
tss          = 0xffffffff84018434
curvnet      = 0xfffff80005383440

Crashing repeatedly. Multiple crash reports available.

Also see: https://netgate.slack.com/archives/C4GUL8CKF/p1671555158444819
Similar crash in 22.05.

Actions #1

Updated by Steve Wheeler 8 months ago

  • Subject changed from Kernal panic in HA nodes when under high load to Kernel panic in HA nodes when under high load
Actions #2

Updated by Kristof Provost 7 months ago

  • Assignee set to Kristof Provost

The backtrace address shows we're crashing in `if (PF_ANEQ(pd->src, &nk->addr[pd->sidx], pd->af) ||`. That likely means that one or both of the state keys were NULL.
Those get set to NULL when the state is being detached, in pf_detach_state(). That's done with the key lock held, but doesn't hold the id lock.

That doesn't automatically mean that pf_test_state_udp() could be processing a state while pf_detach_state() is removing the keys however, as pf_unlink_state() sets the state timeout to PFTM_UNLINKED, which causes pf_find_state() to pretend the state doesn't exist. In other words, once pf_unlink_state() is called on a state we should not find the state in pf_test_state_udp() any more.,

There are two scenarios where this could still happen, and that's when the state creation fails. pf_insert_state() can also call pf_detach_state(), but does not set the timeout to PFTM_UNLINKED, the same is true for pf_state_key_attach(), so perhaps we wound up in one of those error paths.
That would leave a (brief) window where other threads could find the state (because timeout != PFTM_UNLINKED) and acquire a state lock, while simultaneously pf_detach_state() sets the keys to NULL, leading to the above crash.

That's all speculative, because there's really not enough information here to be sure. Still, if that really is the cause there's a very simple and low risk patch to fix it: https://reviews.freebsd.org/D45101

Actions #3

Updated by Jim Pingle 7 months ago

  • Target version set to 2.8.0
  • Plus Target Version set to 24.07
Actions #4

Updated by Kristof Provost 7 months ago

  • Status changed from New to Feedback

What is hoped to be the fix has been merged to our branches.

Actions #5

Updated by Jim Pingle 7 months ago

  • Plus Target Version changed from 24.07 to 24.08
Actions #6

Updated by Jim Pingle about 2 months ago

  • Plus Target Version changed from 24.08 to 24.11
Actions #7

Updated by Jim Pingle 29 days ago

  • Status changed from Feedback to Resolved

Patch has been in for a while and there have been public builds since. No further reports and no reports of other regressions, so I think we can call this solved for now. If someone hits the same panic again we can reopen/retarget the issue.

Actions

Also available in: Atom PDF