pfSense

To add additional context that may aid in diagnostics:

• The issue presents with any change in WAN interface status - virtual or physical (eg unplug event)
• This can include a PPPoE tunnel to an ISP being bounced or refreshed by the ISP
• The example back-traces above were triggered by the PPPoE WAN link being 'disconnected' whilst the physical interface is still up
• The issue can be replicated by 'disconnecting' the established PPPoE link via the GUI button on Status / Interfaces
• The issue is intermittent and is triggered >50% of the time when under test

Full logs available, on request.

All the above crashes are in ipv6 code, most likely racing against an interface and/or address removal.

Given your description though I have to ask if you are explicitly using ipv6 for anything -- it may be a temporary workaround is to not set ipv6 addresses on any of the interfaces.

The addresses in both the ip6_output() and in6_selecthlim() panics suggest that fib6_lookup() returned an nhop_object with a struct ifnet where if_afdata[AF_INET6] is NULL.
That shouldn't happen, and it's not clear to me how it can happen.

I've attempted to replicate this setup, with PPPoE carrying IPv6 traffic (iperf3 --bidir -P 4, with GUA and LL addresses) and hitting the 'Disconnect WAN' button on the Status/Interfaces page. Over 10+ attempts the system remained up, returned errors to iperf3 and re-connected without issue (when triggered via the status page).

I should add that I've been running iperf3 on the pfsense device. The backtraces show locally originated traffic, so that seemed like the setup most likely to trigger this issue.

Mateusz Guzik wrote in #note-3:

All the above crashes are in ipv6 code, most likely racing against an interface and/or address removal.

Given your description though I have to ask if you are explicitly using ipv6 for anything -- it may be a temporary workaround is to not set ipv6 addresses on any of the interfaces.

In my specific use-case yes, I have equipment that I link to which are pure IPv6.

I run an exceptionally common config for the UK market. It is a standard IPv4 and IPv6 WAN connection over PPPoE to the main UK backbone (Openreach). The PPPoE MTU is a little higher than in some counties as it runs a standard 1500 MTU through the tunnel, with the physical link running a bit higher (ie a minimum of 1508 MTU) to absorb the typical PPPoE 8-byte overhead. In my case it is FTTP via an ONT running a 1000/115 Mbit service.

I can run additional tests or diagnostics once outside of normal business hours. Just point me where I need to go.

This may or may not be irrelevant to the underlying fault but combing through other logs I can multiple WAN PPPoE connection attempts and failures. The ppp.log frag below is a successful PPPoE handshake. Even here I can see some hiccups; if I substitute a different router the PPPoE connection logs are clean, each and every time.

ShowHide

May 28 15:18:46 Router-8 ppp[17470]: Multi-link PPP daemon for FreeBSD
May 28 15:18:46 Router-8 ppp[17470]:  
May 28 15:18:46 Router-8 ppp[17470]: process 17470 started, version 5.9
May 28 15:18:46 Router-8 ppp[17470]: web: web is not running
May 28 15:18:46 Router-8 ppp[17470]: [wan] Bundle: Interface ng0 created
May 28 15:18:46 Router-8 ppp[17470]: [undefined] GetSystemIfaceMTU: SIOCGIFMTU failed: Device not configured
May 28 15:18:46 Router-8 ppp[17470]: [wan_link0] Link: OPEN event
May 28 15:18:46 Router-8 ppp[17470]: [wan_link0] LCP: Open event
May 28 15:18:46 Router-8 ppp[17470]: [wan_link0] LCP: state change Initial --> Starting
May 28 15:18:46 Router-8 ppp[17470]: [wan_link0] LCP: LayerStart
May 28 15:18:46 Router-8 ppp[17470]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 28 15:18:46 Router-8 ppp[17470]: [wan_link0] PPPoE: Connecting to ''
May 28 15:18:48 Router-8 ppp[17470]: PPPoE: rec'd ACNAME "acc-aln2.gb-tny" 
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] PPPoE: rec'd PPP-Max-Payload '1500'
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] PPPoE: connection successful
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] Link: UP event
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: Up event
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: state change Starting --> Req-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: SendConfigReq #1
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   PROTOCOMP
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xe9564b72
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: rec'd Configure Request #66 (Req-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   AUTHPROTO CHAP MD5
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0x4edf4b0e
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: SendConfigAck #66
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   AUTHPROTO CHAP MD5
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0x4ddf6b0e
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: state change Req-Sent --> Ack-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: rec'd Configure Reject #1 (Ack-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   PROTOCOMP
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: SendConfigReq #2
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xe9864c72
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: rec'd Configure Ack #2 (Ack-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xe9774c72
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: state change Ack-Sent --> Opened
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: auth: peer wants CHAP, I want nothing
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: LayerUp
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: rec'd CHALLENGE #1 len: 71
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   Name: "acc-aln2.gb-tny" 
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: Using authname "redacted@idnet" 
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: sending RESPONSE #1 len: 37
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: rec'd Configure Request #1 (Opened)
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   AUTHPROTO CHAP MD5
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xf7c1e3e8
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: LayerDown
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: SendConfigReq #3
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xe9763c72
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: SendConfigAck #1
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   AUTHPROTO CHAP MD5
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xf7b2e3e8
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: state change Opened --> Ack-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: rec'd Configure Ack #3 (Ack-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MRU 1500
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   MAGICNUM 0xe9754c72
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: state change Ack-Sent --> Opened
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: auth: peer wants CHAP, I want nothing
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: LayerUp
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: rec'd CHALLENGE #1 len: 31
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0]   Name: "cor2.lond2" 
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: Using authname "redacted@idnet" 
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: sending RESPONSE #1 len: 37
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] CHAP: rec'd SUCCESS #1 len: 4
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] LCP: authorization successful
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] Link: Matched action 'bundle "wan" ""'
May 28 15:18:48 Router-8 ppp[17470]: [wan_link0] Link: Join bundle "wan" 
May 28 15:18:48 Router-8 ppp[17470]: [wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: Open event
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: state change Initial --> Starting
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: LayerStart
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: Open event
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: state change Initial --> Starting
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: LayerStart
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: Up event
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: state change Starting --> Req-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: SendConfigReq #1
May 28 15:18:48 Router-8 ppp[17470]: [wan]   IPADDR 0.0.0.0
May 28 15:18:48 Router-8 ppp[17470]: [wan]   COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: Up event
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: state change Starting --> Req-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: SendConfigReq #1
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: rec'd Configure Request #1 (Req-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan]   IPADDR re.da.ct.ed
May 28 15:18:48 Router-8 ppp[17470]: [wan]     re.da.ct.ed is OK
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: SendConfigAck #1
May 28 15:18:48 Router-8 ppp[17470]: [wan]   IPADDR re.da.ct.ed
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: state change Req-Sent --> Ack-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: rec'd Configure Request #1 (Req-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: SendConfigAck #1
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: state change Req-Sent --> Ack-Sent
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: rec'd Configure Reject #1 (Ack-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan]   COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPCP: SendConfigReq #2
May 28 15:18:48 Router-8 ppp[17470]: [wan]   IPADDR 0.0.0.0
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: rec'd Configure Ack #1 (Ack-Sent)
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: state change Ack-Sent --> Opened
May 28 15:18:48 Router-8 ppp[17470]: [wan] IPV6CP: LayerUp
May 28 15:18:48 Router-8 ppp[17470]: [wan]   reda:cted:reda:cted -> reda:cted:reda:cted
May 28 15:18:51 Router-8 ppp[17470]: [wan] IFACE: Up event
May 28 15:18:51 Router-8 ppp[17470]: [wan] IFACE: Rename interface ng0 to pppoe0
May 28 15:18:51 Router-8 ppp[17470]: [wan] IFACE: Add description "WAN" 
May 28 15:18:51 Router-8 ppp[17470]: [wan] IPCP: rec'd Configure Nak #2 (Ack-Sent)
May 28 15:18:51 Router-8 ppp[17470]: [wan]   IPADDR re.da.ct.ed
May 28 15:18:51 Router-8 ppp[17470]: [wan]     re.da.ct.ed is OK
May 28 15:18:51 Router-8 ppp[17470]: [wan] IPCP: SendConfigReq #3
May 28 15:18:51 Router-8 ppp[17470]: [wan]   IPADDR re.da.ct.ed
May 28 15:18:51 Router-8 ppp[17470]: [wan] IPCP: rec'd Configure Ack #3 (Ack-Sent)
May 28 15:18:51 Router-8 ppp[17470]: [wan]   IPADDR re.da.ct.ed
May 28 15:18:51 Router-8 ppp[17470]: [wan] IPCP: state change Ack-Sent --> Opened
May 28 15:18:51 Router-8 ppp[17470]: [wan] IPCP: LayerUp
May 28 15:18:51 Router-8 ppp[17470]: [wan]   re.da.ct.ed -> re.da.ct.ed
[23.05-RELEASE][admin@Router-8.*******.me]/root: 

☕️

Kristof Provost wrote in #note-4:

The addresses in both the ip6_output() and in6_selecthlim() panics suggest that fib6_lookup() returned an nhop_object with a struct ifnet where if_afdata[AF_INET6] is NULL.
That shouldn't happen, and it's not clear to me how it can happen.

This is probably a race where you get far enough that any sanity checks already passed, but nothing guaranteed the already checked conditions remain stable for the duration and if you are unlucky enough they did change late enough.

Rob A wrote in #note-6:

Mateusz Guzik wrote in #note-3:

All the above crashes are in ipv6 code, most likely racing against an interface and/or address removal.

Given your description though I have to ask if you are explicitly using ipv6 for anything -- it may be a temporary workaround is to not set ipv6 addresses on any of the interfaces.

In my specific use-case yes, I have equipment that I link to which are pure IPv6.

Thanks, I'll look into it. This is most likely either a 1h job OR several days, no in-between.

PPPoE reconnection WITHOUT triggering a pfSense Crash

From the 2am time slot this looks like an ISP-triggered reconnection and it did not cause pfSense to crash. I have added the ppp.log from this otherwise 'non-event' as it captures the multiple failed attempts to complete a full PPPoE handshake. I have never experienced anything other than a first-time connection with my other non-Netgate/pfSense router hardware. Perhaps these failed and repeated attempts are part of the issue?

ShowHide

May 28 15:18:51 Router-8 ppp[17470]: [wan]   93.00.000.00 -> 212.00.000.00
May 31 01:59:12 Router-8 ppp[17470]: caught fatal signal TERM
May 31 01:59:12 Router-8 ppp[17470]: [wan] IFACE: Close event
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPCP: Close event
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPCP: state change Opened --> Closing
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPCP: SendTerminateReq #4
May 31 01:59:12 Router-8 ppp[17470]: [wan] error writing len 8 frame to b0: Network is down
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPCP: LayerDown
May 31 01:59:12 Router-8 ppp[17470]: [wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPV6CP: Close event
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPV6CP: state change Opened --> Closing
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPV6CP: SendTerminateReq #2
May 31 01:59:12 Router-8 ppp[17470]: [wan] error writing len 8 frame to b0: Network is down
May 31 01:59:12 Router-8 ppp[17470]: [wan] IPV6CP: LayerDown
May 31 01:59:14 Router-8 ppp[17470]: [wan] IFACE: Down event
May 31 01:59:14 Router-8 ppp[17470]: [wan] IFACE: Rename interface pppoe0 to pppoe0
May 31 01:59:14 Router-8 ppp[17470]: [wan] IFACE: Set description "WAN" 
May 31 01:59:15 Router-8 ppp[17470]: [wan] IPV6CP: SendTerminateReq #3
May 31 01:59:15 Router-8 ppp[17470]: [wan] error writing len 8 frame to b0: Network is down
May 31 01:59:15 Router-8 ppp[17470]: [wan] IPCP: SendTerminateReq #5
May 31 01:59:15 Router-8 ppp[17470]: [wan] error writing len 8 frame to b0: Network is down
May 31 01:59:16 Router-8 ppp[17470]: [wan] Bundle: Shutdown
May 31 01:59:16 Router-8 ppp[17470]: [wan_link0] Link: Shutdown
May 31 01:59:16 Router-8 ppp[17470]: process 17470 terminated
May 31 01:59:21 Router-8 ppp[77250]: Multi-link PPP daemon for FreeBSD
May 31 01:59:21 Router-8 ppp[77250]:  
May 31 01:59:21 Router-8 ppp[77250]: process 77250 started, version 5.9
May 31 01:59:21 Router-8 ppp[77250]: web: web is not running
May 31 01:59:21 Router-8 ppp[77250]: [wan] Bundle: Interface ng0 created
May 31 01:59:21 Router-8 ppp[77250]: [undefined] GetSystemIfaceMTU: SIOCGIFMTU failed: Device not configured
May 31 01:59:21 Router-8 ppp[77250]: [wan_link0] Link: OPEN event
May 31 01:59:21 Router-8 ppp[77250]: [wan_link0] LCP: Open event
May 31 01:59:21 Router-8 ppp[77250]: [wan_link0] LCP: state change Initial --> Starting
May 31 01:59:21 Router-8 ppp[77250]: [wan_link0] LCP: LayerStart
May 31 01:59:21 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 01:59:21 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 01:59:30 Router-8 ppp[77250]: [wan_link0] PPPoE connection timeout after 9 seconds
May 31 01:59:30 Router-8 ppp[77250]: [wan_link0] Link: DOWN event
May 31 01:59:30 Router-8 ppp[77250]: [wan_link0] LCP: Down event
May 31 01:59:30 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 1 in 4 seconds
May 31 01:59:34 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 1
May 31 01:59:34 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 01:59:34 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 01:59:43 Router-8 ppp[77250]: [wan_link0] PPPoE connection timeout after 9 seconds
May 31 01:59:43 Router-8 ppp[77250]: [wan_link0] Link: DOWN event
May 31 01:59:43 Router-8 ppp[77250]: [wan_link0] LCP: Down event
May 31 01:59:43 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 2 in 4 seconds
May 31 01:59:47 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 2
May 31 01:59:47 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 01:59:47 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 01:59:56 Router-8 ppp[77250]: [wan_link0] PPPoE connection timeout after 9 seconds
May 31 01:59:56 Router-8 ppp[77250]: [wan_link0] Link: DOWN event
May 31 01:59:56 Router-8 ppp[77250]: [wan_link0] LCP: Down event
May 31 01:59:56 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 3 in 2 seconds
May 31 01:59:58 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 3
May 31 01:59:58 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 01:59:58 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 02:00:07 Router-8 ppp[77250]: [wan_link0] PPPoE connection timeout after 9 seconds
May 31 02:00:07 Router-8 ppp[77250]: [wan_link0] Link: DOWN event
May 31 02:00:07 Router-8 ppp[77250]: [wan_link0] LCP: Down event
May 31 02:00:07 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 4 in 1 seconds
May 31 02:00:08 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 4
May 31 02:00:08 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 02:00:08 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 02:00:17 Router-8 ppp[77250]: [wan_link0] PPPoE connection timeout after 9 seconds
May 31 02:00:17 Router-8 ppp[77250]: [wan_link0] Link: DOWN event
May 31 02:00:17 Router-8 ppp[77250]: [wan_link0] LCP: Down event
May 31 02:00:17 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 5 in 2 seconds
May 31 02:00:19 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 5
May 31 02:00:19 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 02:00:19 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 02:00:28 Router-8 ppp[77250]: [wan_link0] PPPoE connection timeout after 9 seconds
May 31 02:00:28 Router-8 ppp[77250]: [wan_link0] Link: DOWN event
May 31 02:00:28 Router-8 ppp[77250]: [wan_link0] LCP: Down event
May 31 02:00:28 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 6 in 1 seconds
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] Link: reconnection attempt 6
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] PPPoE: Connecting to ''
May 31 02:00:29 Router-8 ppp[77250]: PPPoE: rec'd ACNAME "redacted" 
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] PPPoE: rec'd PPP-Max-Payload '1500'
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] PPPoE: connection successful
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] Link: UP event
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: Up event
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: state change Starting --> Req-Sent
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: SendConfigReq #1
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   PROTOCOMP
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x976afadc
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: rec'd Configure Request #144 (Req-Sent)
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   AUTHPROTO CHAP MD5
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x7d183dd2
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: SendConfigAck #144
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   AUTHPROTO CHAP MD5
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x7d183dd2
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: state change Req-Sent --> Ack-Sent
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: rec'd Configure Reject #1 (Ack-Sent)
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   PROTOCOMP
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: SendConfigReq #2
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x976afadc
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: rec'd Configure Ack #2 (Ack-Sent)
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x976afadc
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: state change Ack-Sent --> Opened
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: auth: peer wants CHAP, I want nothing
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: LayerUp
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] CHAP: rec'd CHALLENGE #1 len: 60
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   Name: "redacted" 
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] CHAP: Using authname "redacted@idnet" 
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] CHAP: sending RESPONSE #1 len: 37
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: rec'd Configure Request #1 (Opened)
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   AUTHPROTO CHAP MD5
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x8bb5e796
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: LayerDown
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: SendConfigReq #3
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x976afadc
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: SendConfigAck #1
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   AUTHPROTO CHAP MD5
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x8bb5e796
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: state change Opened --> Ack-Sent
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: rec'd Configure Ack #3 (Ack-Sent)
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MRU 1500
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   MAGICNUM 0x976afadc
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: state change Ack-Sent --> Opened
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: auth: peer wants CHAP, I want nothing
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] LCP: LayerUp
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] CHAP: rec'd CHALLENGE #1 len: 31
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0]   Name: "redacted" 
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] CHAP: Using authname "*******@idnet" 
May 31 02:00:29 Router-8 ppp[77250]: [wan_link0] CHAP: sending RESPONSE #1 len: 37
May 31 02:00:30 Router-8 ppp[77250]: [wan_link0] CHAP: rec'd SUCCESS #1 len: 4
May 31 02:00:30 Router-8 ppp[77250]: [wan_link0] LCP: authorization successful
May 31 02:00:30 Router-8 ppp[77250]: [wan_link0] Link: Matched action 'bundle "wan" ""'
May 31 02:00:30 Router-8 ppp[77250]: [wan_link0] Link: Join bundle "wan" 
May 31 02:00:30 Router-8 ppp[77250]: [wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: Open event
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: state change Initial --> Starting
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: LayerStart
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: Open event
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: state change Initial --> Starting
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: LayerStart
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: Up event
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: state change Starting --> Req-Sent
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: SendConfigReq #1
May 31 02:00:30 Router-8 ppp[77250]: [wan]   IPADDR 0.0.0.0
May 31 02:00:30 Router-8 ppp[77250]: [wan]   COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: Up event
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: state change Starting --> Req-Sent
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: SendConfigReq #1
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: rec'd Configure Request #1 (Req-Sent)
May 31 02:00:30 Router-8 ppp[77250]: [wan]   IPADDR 212.00.000.00
May 31 02:00:30 Router-8 ppp[77250]: [wan]     212.00.000.00 is OK
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: SendConfigAck #1
May 31 02:00:30 Router-8 ppp[77250]: [wan]   IPADDR 212.00.000.00
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: state change Req-Sent --> Ack-Sent
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: rec'd Configure Request #1 (Req-Sent)
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: SendConfigAck #1
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: state change Req-Sent --> Ack-Sent
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: rec'd Configure Reject #1 (Ack-Sent)
May 31 02:00:30 Router-8 ppp[77250]: [wan]   COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPCP: SendConfigReq #2
May 31 02:00:30 Router-8 ppp[77250]: [wan]   IPADDR 0.0.0.0
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: rec'd Configure Ack #1 (Ack-Sent)
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: state change Ack-Sent --> Opened
May 31 02:00:30 Router-8 ppp[77250]: [wan] IPV6CP: LayerUp
May 31 02:00:30 Router-8 ppp[77250]: [wan]   reda:cted:reda:cted -> reda:cted:reda:cted
May 31 02:00:32 Router-8 ppp[77250]: [wan] IFACE: Up event
May 31 02:00:32 Router-8 ppp[77250]: [wan] IFACE: Rename interface ng0 to pppoe0
May 31 02:00:32 Router-8 ppp[77250]: [wan] IFACE: Add description "WAN" 
May 31 02:00:32 Router-8 ppp[77250]: [wan] IPCP: rec'd Configure Nak #2 (Ack-Sent)
May 31 02:00:32 Router-8 ppp[77250]: [wan]   IPADDR 93.00.000.00
May 31 02:00:32 Router-8 ppp[77250]: [wan]     93.00.000.00 is OK
May 31 02:00:32 Router-8 ppp[77250]: [wan] IPCP: SendConfigReq #3
May 31 02:00:32 Router-8 ppp[77250]: [wan]   IPADDR 93.00.000.00
May 31 02:00:32 Router-8 ppp[77250]: [wan] IPCP: rec'd Configure Ack #3 (Ack-Sent)
May 31 02:00:32 Router-8 ppp[77250]: [wan]   IPADDR 93.00.000.00
May 31 02:00:32 Router-8 ppp[77250]: [wan] IPCP: state change Ack-Sent --> Opened
May 31 02:00:32 Router-8 ppp[77250]: [wan] IPCP: LayerUp
May 31 02:00:32 Router-8 ppp[77250]: [wan]   93.00.000.00 -> 212.00.000.00
[23.05-RELEASE][admin@Router-8.*******.me]/root: 

[Is there a code or button to add a spoiler type of truncation to logs such as this, as they break-up the readability of the thread somewhat?]

Found it, the collapse code. I will try harder, just getting used to Redmine.

☕️

After poking around here is my analysis, which confirms my preliminary suspicion:

All of the crash sites are invoking if_getafdata(ifp, AF_INET6), typically through the ND_IFINFO macro. The routine returns a pointer to protocol-specific if_data, which when crashing happens to be NULL.

It stems from a race against interface destruction, which eventually does:

static int
if_detach_internal(struct ifnet *ifp, bool vmove)
{
[...]
        IF_AFDATA_LOCK(ifp);
        i = ifp->if_afdata_initialized;
        ifp->if_afdata_initialized = 0;
        IF_AFDATA_UNLOCK(ifp);
        if (i == 0)
                return (0);
        SLIST_FOREACH(dp, &domains, dom_next) {
                if (dp->dom_ifdetach && ifp->if_afdata[dp->dom_family]) {
                        (*dp->dom_ifdetach)(ifp,
                            ifp->if_afdata[dp->dom_family]); // uaf after this
                        ifp->if_afdata[dp->dom_family] = NULL; // NULL pointer deref after this
                }
        }
[...]

As ipv6 support has a dom_ifdetach, this results in a use-after-free or a NULL pointer dereference, depending on timing of parallel access.

The entire thing is rather seriously misdesigned, but as a damage-controlling measure I think one can split dom_ifdetach into two parts, where the last one only executes when the interface itself is going away. I'm going to have to flame this over with network people.

Is Tailscale also in play here? I've trying and failing to reproduce this again. No matter what I try to do, I simply cannot get the system to panic. Clearly I'm missing some factor, but right now I have no idea what that'd be.

In my case there is no involvement of Tailscale as I do not use it.

Regards.

☕️

I have switched to 23.09 dev as that is where most of the activity is focused. I will monitor and update if this issue has moved across to 23.09 dev or not.

☕️

Issue remains 'live' with 23.09 dev. Details of the first crash on this version, triggered this time by taking the WAN interface down and then up via the GUI:

Crash report begins.  Anonymous machine information:

amd64
14.0-ALPHA2
FreeBSD 14.0-ALPHA2 amd64 1400094 #1 plus-devel-main-n256133-bef8dca4536: Tue Sep  5 06:26:19 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/obj/amd64/fWgcJpOQ/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/s

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/nda0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 228864
  Blocksize: 512
  Compression: none
  Dumptime: 2023-09-06 18:00:47 +0100
  Hostname: Router-8.redacted.me
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.0-ALPHA2 amd64 1400094 #1 plus-devel-main-n256133-bef8dca4536: Tue Sep  5 06:26:19 UTC 2023
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/obj/amd64/fW
  Panic String: page fault
  Dump Parity: 954902300
  Bounds: 0
  Dump Status: good

db:1:pfs> bt
Tracing pid 2 tid 100041 td 0xfffffe0085272560
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850c5840
vpanic() at vpanic+0x163/frame 0xfffffe00850c5970
panic() at panic+0x43/frame 0xfffffe00850c59d0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00850c5a30
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850c5a90
calltrap() at calltrap+0x8/frame 0xfffffe00850c5a90
--- trap 0xc, rip = 0xffffffff80f4d9e6, rsp = 0xfffffe00850c5b60, rbp = 0xfffffe00850c5b90 ---
in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850c5b90
tcp_default_output() at tcp_default_output+0x1d97/frame 0xfffffe00850c5d70
tcp_timer_rexmt() at tcp_timer_rexmt+0x52f/frame 0xfffffe00850c5dd0
tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe00850c5e10
softclock_call_cc() at softclock_call_cc+0x134/frame 0xfffffe00850c5ec0
softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850c5ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00850c5f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850c5f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Full logs available on request.

Core dump provided to Christian McDonald for the related ndp issue.

☕️

Understood and thanks for the heads-up that the fix may be 6 months away. I'll have to find a new router solution in the interim, meaning different hardware too as I run pfSense on a Netgate 6100. 🤕

Standing-by as a test-bed in the interim period.

☕️

Unfortunately both Steve and I have been unable to reproduce this problem.

We could try to see if a full core dumps helps, but I suspect this to be a race condition and even the core file might not help.

Still, if you're game there are instructions on how to get a full core dump here: https://forum.netgate.com/post/1127502
We also need that to be done with a kernel we have debug symbols for. Can you install the latest snapshot kernel-debug (which includes symbols files)? Or I can provide a kernel I have symbols for, if that's easier.

Thanks Kristof, as it happens I had a crash today:

db:1:pfs> bt
Tracing pid 2 tid 100043 td 0xfffffe0085271720
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850bb840
vpanic() at vpanic+0x163/frame 0xfffffe00850bb970
panic() at panic+0x43/frame 0xfffffe00850bb9d0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00850bba30
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850bba90
calltrap() at calltrap+0x8/frame 0xfffffe00850bba90
--- trap 0xc, rip = 0xffffffff80f4e116, rsp = 0xfffffe00850bbb60, rbp = 0xfffffe00850bbb90 ---
in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850bbb90
tcp_default_output() at tcp_default_output+0x1d97/frame 0xfffffe00850bbd70
tcp_timer_rexmt() at tcp_timer_rexmt+0x52f/frame 0xfffffe00850bbdd0
tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe00850bbe10
softclock_call_cc() at softclock_call_cc+0x134/frame 0xfffffe00850bbec0
softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850bbef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00850bbf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850bbf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:1:pfs>  show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx         0xffffffff81457767
rdx                      0x3f8
rbx                      0x100
rsp         0xfffffe00850bb840
rbp         0xfffffe00850bb840
rsi                 0xc75d684a
rdi                        0x4
r8                0xb3c75d684a
r9          0xfffffe0085271720
r10         0xfffffe00850bb720
r11         0xcedfc2df9afff59c
r12                          0
r13                          0
r14         0xffffffff813db3a9
r15         0xfffffe0085271720
rip         0xffffffff80d388c2  kdb_enter+0x32
rflags                    0x86
kdb_enter+0x32: movq    $0,0x2344f43(%rip)
db:1:pfs>......

It does have the feel of a race-condition and with Steve stuck on G.fast he is probably sheltered from it.

Happy to provide a kernel dump, just in case it helps. I didn't quite understand the instructions as part of that thread though. I have managed one previously where Christian provided everything I needed - presumably the 'kernel + symbols' you mentioned. I seemed to follow those instructions ok and emailed the results to him (ndp issue) https://redmine.pfsense.org/issues/14755#change-69465. So if it can be simplified I can run tests over this weekend. I'm running the latest 23.09d but I have a snapshot back to 23.05.1 if required.

Can you install and run this kernel and try to get a core dump?
https://www.codepro.be/files/pfSense-kernel-debug-pfSense-23.09.a.20231005.1555.pkg

No problem. Best method to install this in a recoverable way?

You can just pkg install and pkg remove it later.
As usual, make a config backup just in case, but this ought to be pretty safe. Make a boot environment to be extra super-duper safe and for very convenient revert later.

No joy. With pkg install I get the error:

pkg: No packages available to install matching 'pfSense-23.09.a.20231005.1555.pkg' have been found in the repositories

Tried with the full url, just the package name and from a local copy of the file only to get a variation of the error above in the CLI.

Using the GUI I loaded it via System Patches... not sure if that is a genuine method or not.

I ran the system (In whatever state I achieved above) but I was fighting other issues such as Kea and pfBlocker not running. Kea could usually be relaunched but the system was a bit buggy. I rolled back to my snapshot to await whatever I did wrong with the pkg.

Download the file to your device, and install with `pkg install -U <filename>`, via the device CLI.

Ok, your job would be easy if it wasn't for these dull customers!

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: pkg install -U /tmp/pfSense-kernel-debug-pfSense-23.09.a.20231005.1555.pkg
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    pfSense-kernel-debug-pfSense: 23.09.a.20231005.1555 [unknown-repository]

Number of packages to be installed: 1

The process will require 305 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing pfSense-kernel-debug-pfSense-23.09.a.20231005.1555...
Extracting pfSense-kernel-debug-pfSense-23.09.a.20231005.1555: 100%
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: 

☕️

With the debug kernel running I triggered a crash and have the regular crash report. I did not see a core dump file in root though and could not find it. Where would the core dump be located?

It should end up in /var/crash

Regrettably no:

/root: ls /var/crash
bounds            info.0            info.last         textdump.tar.0    textdump.tar.last

Did you edit the pfSense-ddb.conf file and add a swap partition for it to dump to?
Christian is working on a shiny webui button to make this all easier, but sadly that’s not ready yet.

Not sure if I have done so previously. Currently the file looks like this:

# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``ddb'' to define debugging scripts.
#
# see ``man 4 ddb'' and ``man 8 ddb'' for details.
#

script lockinfo=show locks; show alllocks; show lockedvnods
script pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace

# kdb.enter.panic       panic(9) was called.
script kdb.enter.default=textdump set; capture on; run pfs ; capture off; textdump dump; reset

# kdb.enter.witness    witness(4) detected a locking error.
script kdb.enter.witness=run lockinfo

To clarify, make sure that after installing the kernel-debug package, you reboot and select the debug kernel (option 6) in the boot loader. Then follow the steps
on the link provided previously:

Short version: add a device for a swap partition, ideally at least as large as system RAM. A USB stick should work. (Note you'll lose all data on the stick!)
If the USB (or other) swap device is da0 do:

gpart destroy -F da0
gpart create -s gpt da0
gpart add -t freebsd-swap da0

Add /dev/da0p1 none swap sw 0 0 to /etc/fstab.
Edit /etc/pfSense-ddb.conf and change the script kdb.enter.default to script kdb.enter.default=bt ; show registers ; dump ; reset.

Reboot.

I'm finding it hard to distinguish 'the steps' in that thread from the normal noise and I don't know how to enter the boot loader for option 6 either.

Perhaps I am in too deep for my level as I am struggling with the 'assumed knowledge' that I don't seem to have when following this issue thread. It felt easier when I was asked for a ndp.core dump last time around.

I'll message you on the forum.

Hopefully I have captured the panic:

<118>Netgate pfSense Plus 23.09-BETA amd64 20231020-0600
<118>Bootup complete
<6>ng0: changing name to 'pppoe0'
pf_test6: kif == NULL, if_xname pppoe0
<6>ng0: changing name to 'pppoe0'

Fatal trap 12: page fault while in kernel mode

cpuid = 3; apic id = 18
fault virtual address    = 0x10
fault code        = supervisor read data, page not present
instruction pointer    = 0x20:0xffffffff80f4e116
stack pointer            = 0x0:0xfffffe00850b6b60
frame pointer            = 0x0:0xfffffe00850b6b90
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 2 (clock (3))
rdi: fffff80203712800 rsi: 000000000000001c rdx: fffff8013760d878
rcx: fffff8013760d878  r8: 00000000ffffffbd  r9: 0000000000000018
rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe00850b6b90
r10: fffff802033dd8c0 r11: fffff8016f5e5000 r12: 0000000000010300
r13: fffff80203676b98 r14: fffffe00850b6b68 r15: 0000000000000018
trap number        = 12
panic: page fault
cpuid = 3
time = 1697905286
KDB: enter: panic

vmcore dump available on request ~ 800 MB.

Dump header from device: /dev/nda0p3
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 791597056
  Blocksize: 512
  Compression: none
  Dumptime: 2023-10-21 17:21:26 +0100
  Hostname: Router-8
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09-n256155-ccf460c6e58: Fri Oct 20 06:38:47 UTC 2023
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/amd64/
  Panic String: page fault
  Dump Parity: 2695527028
  Bounds: 0
  Dump Status: good

The backtrace there shows us in the TCP output path (from a timer callback), sending a packet out. This requires setting the hop limit via `in6_selecthlim(inp, NULL);`.
gdb tries to mislead us a bit with the line number (in that it points to sys/netinet6/in6_src.c:850, which is the `if (ifp)` case, but we've supplied a NULL ifp), but the disassembly points the third case, where we look up the output interface via fib6_lookup(). That returns an ifp, and in this core dump we see an nhob_object pointing to a struct ifnet with various ifdead_* pointers. This would support the theory that we've somehow wound up getting a route via a destroyed interface.

I would expect such routes to be removed as part of the interface destruction, but clearly that's not happening for some reason.

Before if_detach_internal() clears the if_afdata array it also removes all of the addresses from the interface, which ought to ensure we can't return this interface in a route lookup any more.
The address removal is passed to the rib code, which ought to result in the fib also clearing it. I wonder if there's an issue synchronising those. I've not yet been able to dump either rib or fib from this core file, so this is currently an unsupported theory.

Ok, I'll endeavour to get another core dump for you; although it may be as late as the weekend.

I don't expect another core dump to make any difference in that analysis. I may or may not be able to find the information I need in the one I have, but another one is not likely to help.

Understood - thank you.

Spending more time on my Ice Lake Xeon box recently but no observable difference to that of my 6100. Today's crash on 23.09-RC (I will retain the files, in case they offer anything new):

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #0 plus-RELENG_23_09-n256160-b373d8da4a4: Fri Oct 27 02:22:22 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/amd64/ERwoNEbB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/nda0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 301568
  Blocksize: 512
  Compression: none
  Dumptime: 2023-11-04 16:42:38 +0000
  Hostname: Router-7
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.0-CURRENT amd64 1400094 #0 plus-RELENG_23_09-n256160-b373d8da4a4: Fri Oct 27 02:22:22 UTC 2023
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/amd64/
  Panic String: page fault
  Dump Parity: 911556898
  Bounds: 0
  Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������1012407�����14521472176�  7175� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
db:1:pfs> bt
Tracing pid 2 tid 100070 td 0xfffffe00e2238000
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00e1f4a840
vpanic() at vpanic+0x163/frame 0xfffffe00e1f4a970
panic() at panic+0x43/frame 0xfffffe00e1f4a9d0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00e1f4aa30
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00e1f4aa90
calltrap() at calltrap+0x8/frame 0xfffffe00e1f4aa90
--- trap 0xc, rip = 0xffffffff80f4e216, rsp = 0xfffffe00e1f4ab60, rbp = 0xfffffe00e1f4ab90 ---
in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00e1f4ab90
tcp_default_output() at tcp_default_output+0x1d97/frame 0xfffffe00e1f4ad70
tcp_timer_rexmt() at tcp_timer_rexmt+0x52f/frame 0xfffffe00e1f4add0
tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe00e1f4ae10
softclock_call_cc() at softclock_call_cc+0x134/frame 0xfffffe00e1f4aec0
softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00e1f4aef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00e1f4af30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e1f4af30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:1:pfs>  show registers
cs                        0x20
ds                        0x3b
es                        0x3b
fs                        0x13
gs                        0x1b
ss                        0x28
rax                       0x12
rcx         0xffffffff81458a29
rdx         0xfffffe00e1f4a480
rbx                      0x100
rsp         0xfffffe00e1f4a840
rbp         0xfffffe00e1f4a840
rsi                        0xa
rdi         0xffffffff82d4e850  gdb_consdev
r8                           0
r9                           0
r10                          0
r11                          0
r12                          0
r13                          0
r14         0xffffffff813dbf84
r15         0xfffffe00e2238000
rip         0xffffffff80d389c2  kdb_enter+0x32
rflags                    0x86
kdb_enter+0x32: movq    $0,0x2344e43(%rip)
db:1:pfs>  show pcpu
cpuid        = 5
dynamic pcpu = 0xfffffe00b7627f00
curthread    = 0xfffffe00e2238000: pid 2 tid 100070 critnest 1 "clock (5)" 
curpcb       = 0xfffffe00e2238520
fpcurthread  = none
idlethread   = 0xfffffe00e2163000: tid 100008 "idle: cpu5" 
self         = 0xffffffff84015000
curpmap      = 0xffffffff83021ab0
tssp         = 0xffffffff84015384
rsp0         = 0xfffffe00e1f4b000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84015404
ldt          = 0xffffffff84015444
tss          = 0xffffffff84015434
curvnet      = 0xfffff80001306340
db:1:pfs>  run lockinfo
db:2:lockinfo> show locks
No such command; use "help" to list available commands
db:2:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:2:lockinfo>  show lockedvnods
Locked vnodes
db:1:pfs>  acttrace

☕️

I've pushed a workaround to both devel-main and plus-devel-main. That should avoid the panic (or at least make it much less likely to occur), but doesn't fix the underlying issue.