pfSense

I have no idea what could be going on here.

I'm going to assume that the `codel_should_drop: could not found the packet mtag!` warning is related to the panic, but that warning itself ought to be impossible to hit.
It means that there's an mbuf in the codel queue without an MTAG_CODEL. However, I don't see how that can happen, because any insertion into the codel queue is done through codel_addq(), which will either succeed in attaching the mtag, or will drop the packet.

The codel code is presumably called through the altq_priq code, but there too it doesn't add mbufs in any other way.

I've also been unable to reproduce this problem, which itself is odd, given that the original report states this happens consistently at boot. We may be missing a configuration factor.

I believe I've reproduced the problem. It required using ix3 as LAN interface and ix3.201 as PPPoE for WAN.

With a debug kernel I now see this backtrace:

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 0c
instruction pointer     = 0x20:0xffffffff80d8ced0
stack pointer           = 0x28:0xfffffe007c849af0
frame pointer           = 0x28:0xfffffe007c849b10
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 66278 (unbound-anchor)
rdi: deadc0dedeadc0de rsi: fffff800020a2600 rdx: 0000000000000065
rcx: fffff800020a2600  r8: 0000000000000020  r9: 0000000000000004
rax: deadc0dedeadc0de rbx: fffff800031a9b00 rbp: fffffe007c849b10
r10: fffff8000151c738 r11: 0000000000000015 r12: 0000000000000100
r13: fffff800016d9300 r14: fffff800031a9b30 r15: fffff800020a2600
trap number             = 9
panic: general protection fault
cpuid = 0
time = 1687875076
KDB: enter: panic
[ thread pid 66278 tid 100202 ]
Stopped at      kdb_enter+0x32: movq    $0,0x2309603(%rip)
db:0:kdb.enter.default> textdump set
textdump set
db:0:kdb.enter.default>  capture on
db:0:kdb.enter.default>  run pfs
db:1:pfs> bt
Tracing pid 66278 tid 100202 td 0xfffffe007d153560
kdb_enter() at kdb_enter+0x32/frame 0xfffffe007c849910
vpanic() at vpanic+0x181/frame 0xfffffe007c849960
panic() at panic+0x43/frame 0xfffffe007c8499c0
trap_fatal() at trap_fatal+0x409/frame 0xfffffe007c849a20
calltrap() at calltrap+0x8/frame 0xfffffe007c849a20
--- trap 0x9, rip = 0xffffffff80d8ced0, rsp = 0xfffffe007c849af0, rbp = 0xfffffe007c849b10 ---
m_tag_delete_chain() at m_tag_delete_chain+0x40/frame 0xfffffe007c849b10
uma_zfree_arg() at uma_zfree_arg+0xad/frame 0xfffffe007c849b60
m_free() at m_free+0xc3/frame 0xfffffe007c849b90
m_freem() at m_freem+0x28/frame 0xfffffe007c849bb0
iflib_completed_tx_reclaim() at iflib_completed_tx_reclaim+0x216/frame 0xfffffe007c849c30
iflib_txq_drain() at iflib_txq_drain+0x63/frame 0xfffffe007c849cb0
drain_ring_lockless() at drain_ring_lockless+0x9d/frame 0xfffffe007c849d00
ifmp_ring_enqueue() at ifmp_ring_enqueue+0x2aa/frame 0xfffffe007c849d40
iflib_if_transmit() at iflib_if_transmit+0x25e/frame 0xfffffe007c849db0
iflib_altq_if_start() at iflib_altq_if_start+0xde/frame 0xfffffe007c849de0
iflib_altq_if_transmit() at iflib_altq_if_transmit+0xe6/frame 0xfffffe007c849e10
vlan_transmit() at vlan_transmit+0xfc/frame 0xfffffe007c849e60
ether_output_frame() at ether_output_frame+0xa3/frame 0xfffffe007c849e90
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c849f30
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c849f70
ng_pppoe_rcvdata() at ng_pppoe_rcvdata+0x5a9/frame 0xfffffe007c84a000
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c84a0a0
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c84a0e0
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c84a180
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c84a1c0
ng_ppp_link_xmit() at ng_ppp_link_xmit+0x11b/frame 0xfffffe007c84a210
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c84a2b0
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c84a2f0
ng_vjc_rcvdata() at ng_vjc_rcvdata+0x177/frame 0xfffffe007c84a340
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c84a3e0
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c84a420
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c84a4c0
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c84a500
ng_apply_item() at ng_apply_item+0x2de/frame 0xfffffe007c84a5a0
ng_snd_item() at ng_snd_item+0x2e0/frame 0xfffffe007c84a5e0
ng_iface_send() at ng_iface_send+0xf4/frame 0xfffffe007c84a660
ng_iface_start() at ng_iface_start+0x83/frame 0xfffffe007c84a6b0
if_transmit_default() at if_transmit_default+0x13f/frame 0xfffffe007c84a6f0
ng_iface_output() at ng_iface_output+0x1e0/frame 0xfffffe007c84a730
pf_route() at pf_route+0xba0/frame 0xfffffe007c84a810
pf_test() at pf_test+0xe5f/frame 0xfffffe007c84a9c0
pf_check_out() at pf_check_out+0x22/frame 0xfffffe007c84a9e0
pfil_mbuf_out() at pfil_mbuf_out+0x55/frame 0xfffffe007c84aa20
ip_output() at ip_output+0xcd8/frame 0xfffffe007c84ab10
udp_send() at udp_send+0xa18/frame 0xfffffe007c84ac10
sosend_dgram() at sosend_dgram+0x327/frame 0xfffffe007c84ac70
sousrsend() at sousrsend+0x7e/frame 0xfffffe007c84acd0
kern_sendit() at kern_sendit+0x142/frame 0xfffffe007c84ad60
sendit() at sendit+0xb7/frame 0xfffffe007c84adb0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe007c84ae00
amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe007c84af30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe007c84af30
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x82984af2a, rsp = 0x82054ac38, rbp = 0x82054ac80 ---

A bit more progress. It looks like we're enqueuing the same mbuf twice, so it gets used after it's been freed and that's what's causing the panic.

It looks like these are the relevant backtraces:

  0  63710                 codel_addq:entry m = fffff801659fc500
              kernel`priq_enqueue+0x125
              kernel`if_transmit_default+0xb9
              kernel`ng_iface_output+0x1e0
              kernel`pf_route+0xba0
              kernel`pf_test+0xe5f
              kernel`pf_check_out+0x22
              kernel`pfil_mbuf_out+0x55
              kernel`ip_output+0xcd8
              kernel`rip_send+0x460
              kernel`sosend_generic+0x61a
              kernel`sousrsend+0x7e
              kernel`kern_sendit+0x142
              kernel`sendit+0xb7
              kernel`sys_sendto+0x4d
              kernel`amd64_syscall+0x140
              kernel`0xffffffff81292edb

  0  63710                 codel_addq:entry m = fffff801659fc500
              kernel`priq_enqueue+0x125
              kernel`iflib_altq_if_transmit+0xc0
              kernel`vlan_transmit+0xfc
              kernel`ether_output_frame+0xa3
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_pppoe_rcvdata+0x5a9
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_ppp_link_xmit+0x11b
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_vjc_rcvdata+0x177
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_apply_item+0x2de
              kernel`ng_snd_item+0x2e0
              kernel`ng_iface_send+0xf4
              kernel`ng_iface_start+0x83
              kernel`if_transmit_default+0x13f
              kernel`ng_iface_output+0x1e0
              kernel`pf_route+0xba0
              kernel`pf_test+0xe5f
              kernel`pf_check_out+0x22
              kernel`pfil_mbuf_out+0x55
              kernel`ip_output+0xcd8
              kernel`rip_send+0x460
              kernel`sosend_generic+0x61a
              kernel`sousrsend+0x7e
              kernel`kern_sendit+0x142
              kernel`sendit+0xb7
              kernel`sys_sendto+0x4d
              kernel`amd64_syscall+0x140
              kernel`0xffffffff81292edb

I'm looking at ng_iface_start()'s use of IFQ_DRV_DEQUEUE(), because it looks like that may be doing the wrong thing here.

I believe I've identified the cause. It's not quite what I thought initially, but it's close.
We are indeed enqueuing the same mbuf twice, but between the first and second enqueue it does get removed from the queue again, so there's no use-after-free there.

However, the codel code adds an mtag with a timestamp (in codel_addq()):

int
codel_addq(struct codel *c, class_queue_t *q, struct mbuf *m)
{
        struct m_tag *mtag;
        uint64_t *enqueue_time;

        if (qlen(q) < qlimit(q)) {
                mtag = m_tag_locate(m, MTAG_CODEL, 0, NULL);
                if (mtag == NULL)
                        mtag = m_tag_alloc(MTAG_CODEL, 0, sizeof(uint64_t),
                            M_NOWAIT);
                if (mtag == NULL) {
                        m_freem(m);
                        return (-1);
                }
                enqueue_time = (uint64_t *)(mtag + 1);
                *enqueue_time = read_machclk();
                m_tag_prepend(m, mtag);
                _addq(q, m);
                return (0);
        }
        c->drop_overlimit++;
        m_freem(m);

        return (-1);
}

codel_addq() attempts to avoid allocating a second mtag if an MTAG_CODEL tag already exists. That's fine, but it does insert the mtag unconditionally, so if the mbuf already contains an MTAG_CODEL we insert the mtag again. When we get to m_tag_delete_chain() (via m_free()) we delete the same mtag twice, and panic.
I believe the critical elements to seeing this issue are: (1) using ALTQ/codel and (2) doing so over vlan, with a configuration that causes the packet to be handled twice by ALTQ.

I'll spend a few minutes trying to build a test case for this to confirm, but I'm pretty confident that moving the m_tag_prepend() in the same if case as m_tag_alloc() will fix the panic.

Fixed upstream in https://cgit.freebsd.org/src/commit/?id=d0b0424fa0ca8fb239e00d6bdd5e6340b7a85e68
Test case: https://cgit.freebsd.org/src/commit/?id=e8423423737ec0c08fc021d58795d5d34a55dda2