pfSense

I saw a crash with a backtrace like that once on a test VM with an L2TP WAN but only one time, not repeatedly, so I chalked it up to a random power or other VM-specific event.

Approximately how often is "regularly"? Have you noticed any pattern to when it happens? Is it after a certain amount of traffic, certain amount of time, or randomly?

Right now it happens at least once a day, but at random times. I'll check if the amount of traffic might be related.

After a few more crashes with different error messages, I ran a memory test, which showed errors. RAM is replaced and I will report back if the error occurs again, but right now I think this can be closed.

• Time (20:00 & 03:10)
• Uptime (56h & 31h)
• Packout count (10/9M & 7/5M)
• Byte count (4.1/2.5GB & 3.1/1.1GB)

The byte/s and packet/s also have much higher peeks between the crashes.

yes it's always the same (except the hex addresses)

Tracing pid 13 tid 100022 td 0xfffff800039b4000
m_copypacket() at m_copypacket+0x16/frame 0xfffffe01195f7ab0
ng_l2tp_seq_rack_timeout() at ng_l2tp_seq_rack_timeout+0x164/frame 0xfffffe01195f7af0
ng_apply_item() at ng_apply_item+0x8f/frame 0xfffffe01195f7b70
ngthread() at ngthread+0x10a/frame 0xfffffe01195f7bb0
fork_exit() at fork_exit+0x83/frame 0xfffffe01195f7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01195f7bf0

This seems to be an upstream bug in FreeBSD mpd5 - today I got the same crash on my L2TP Server (FreeBSD 11.2-RELEASE-p2 / mpd5 5.8):

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address      = 0x1c
fault code         = supervisor read data, page not present
instruction pointer        = 0x20:0xffffffff80b79f06
stack pointer              = 0x28:0xfffffe0094ab8a60
frame pointer              = 0x28:0xfffffe0094ab8aa0
code segment               = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process            = 619 (ng_queue2)
trap number                = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff80b3d567 at kdb_backtrace+0x67
#1 0xffffffff80af6b07 at vpanic+0x177
#2 0xffffffff80af6983 at panic+0x43
#3 0xffffffff80f77fcf at trap_fatal+0x35f
#4 0xffffffff80f78029 at trap_pfault+0x49
#5 0xffffffff80f777f7 at trap+0x2c7
#6 0xffffffff80f57dac at calltrap+0x8
#7 0xffffffff82835bf4 at ng_l2tp_seq_rack_timeout+0x164
#8 0xffffffff8282868d at ng_apply_item+0xcd
#9 0xffffffff8282b084 at ngthread+0x1a4
#10 0xffffffff80aba083 at fork_exit+0x83
#11 0xffffffff80f58cce at fork_trampoline+0xe

I've opened a bug at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241133

A possible solution proposed by markj@ - https://reviews.freebsd.org/D26548

If this revision is accepted I'll import it into our tree and we can validate the fix

As far as I can tell this has been accepted upstream (https://svnweb.freebsd.org/changeset/base/366167). Can someone point me to a howto or tutorial, which explains, what I need to do to apply this fix to pfsense? I've hit this bug at least twice in the last three days.

I'm currently running pfSense 2.4.5-RELEASE-p1 (amd64).

Bianco Veigel wrote:

I've updated to 2.5.0.a.20200928.1250 and got the same crash as before. I've attached the crashdump.

Is there anything else, I can do?

I've added INVARIANTS to kernel in order to collect more useful data from crash. Please try it again on 20200929-1250 image

https://reviews.freebsd.org/D26586 was also imported to devel-12 branch. Next round of snapshots (1250) will have this applied

I'd like to test the new version, but there is still no 20200929-1250.

Bianco Veigel wrote:

I'd like to test the new version, but there is still no 20200929-1250.

That build failed. It's now fixed and a new image will be available in ~3h

Bianco Veigel wrote:

I'd like to test the new version, but there is still no 20200929-1250.

A new snapshot is available

Bianco Veigel wrote:

it crashed again with 2.5.0.a.20200930.0050

Thanks for your patience so far, it's very appreciated. I have another debugging patch that fixes a few other possible races and prints some extra info. I'll try to get it into a snapshot soon.

Is there anything I can do, to help you fix this? I'm still hitting this bug regularly (up to multiple times per day).

I'm currently running pfSense 2.5.0-RELEASE as l2tp client against FreeBSD 12.2-RELEASE-p7 as l2tp server.

I understand, that this is an unusual setup, so I'm willing to offer bounty of 500€ (either directly paid to the developer or as a donation to the FreeBSD Foundation).

Bianco Veigel wrote in #note-33:

I've upgraded to 2.6.0-DEVELOPMENT built on Wed Aug 04 01:14:35 EDT 2021 and it crashed again. The crash dumps are attached.

Thanks!

A new patch with an attempt to fix it was pushed and will be available on tomorrow's 2.6.0 snapshot.

Hi Bianco,

did you get the chance to test the fix?

If you check dmesg and see messages like these:

ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff800d22c4878
ng_l2tp_seq_rack_timeout; empty transmit queue on seq 0xfffff800d22c4878: ns 725 rack 725 active 2

then the problematic condition occurred.

I've updated to 2.6.0-DEVELOPMENT (amd64) built on Fri Aug 06 01:10:08 EDT 2021 this evening, and am waiting if it crashes again. I will report in the next 7 days, if it crashed again or not.

Sounds good, thanks for the update!

Mateusz Guzik wrote in #note-35:

Hi Bianco,

did you get the chance to test the fix?

If you check dmesg and see messages like these:
[...]

then the problematic condition occurred.

Seems like the problemativ condition occured, but this time it didn't crash:

ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
ng_l2tp_seq_rack_timeout; empty transmit queue on seq 0xfffff8008bd22878: ns 1005 rack 1005 active 2ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878

Now I'm getting the following message every minute:

2021 Aug  7 12:58:06 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 12:59:06 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:00:07 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:01:07 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:02:06 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:03:06 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:04:07 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:05:07 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:06:06 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878
2021 Aug  7 13:07:06 tony.bln.wg1337.de ng_l2tp_seq_recv_nr: rack_timer still active on seq 0xfffff8008bd22878

All the messages will be removed soon. They are only there right now to confirm the problem condition is being exercised.

It was clear to me, that those messages are only for debugging - I was concerned, that the timer is still running for an hour.

PS: I've hit the condition multiple times again today, but my pfSense is still running and the L2TP connection was not interrupted.

I removed debug printfs so updating to upcoming snapshot should stop the spam. I'll wait for your final confirmation that the bug indeed is fixed before altering the ticket.

It's been running stable for five days (which is twice the max uptime of the last 30 days), so I'll say it's fixed.
Please let me know, were to throw my money at.

Glad to hear it is working for you.

The work was sponsored by Netgate, so I suggest you use the money to buy more products :)