pfSense

Quite a few ALIX systems upgraded with no issue along those lines. Is it always after "Generating RRD graphs"? If you haven't tried the upgrade with it set to permanent rw mode under Diag>NanoBSD, that's a good thing to try.

Chris Buechler wrote:

Is it always after "Generating RRD graphs"?

Yes, always there

If you haven't tried the upgrade with it set to permanent rw mode under Diag>NanoBSD, that's a good thing to try.

I did it already, sorry, I forgot to mention. Also, the option "Keep media mounted read/write at all times" is activated.

Thanks!

not sure what to suggest, haven't seen that and went through another ALIX 2.2.6->2.3 upgrade with no issue today.

Do you have any hardware other than the board itself connected? mini-PCI cards, USB devices.

Do you know if that same config has the same end result restored to a different ALIX? If so I'd like to get a config backup

Do you have any hardware other than the board itself connected? mini-PCI cards, USB devices.

Well, the only hardware connected is the Crypto Accelerator in a mini-PCI port (Hardware crypto: Hifn 7955), there is nothing else connected.

Do you know if that same config has the same end result restored to a different ALIX? If so I'd like to get a config backup.

Honestly I have only one ALIX box, but I can send you in PM a full backup, RRD data included if needed.

Same on version 2.3.1 unfortunately. :(

I can confirm the issue on my setup as well, a Negate m1n1wall with pfSense 2.3.3_1 and a Hifn 7955 crypto accelerator card. As soon as traffic is initiated to hosts defined in IPSec Phase 2 entries, Fatal trap 12 occurs and then a lot of trace logs from charon process.

Tracing pid 12 tid 100004 td 0xc4278c80
bcopy(0,d31f9738,18,c07f62e0,c4bcea78,...) at bcopy+0x1a/frame 0xc3ff1798
hifn_process(c41c2a80,c5840bf4,0,c27e0820,0,...) at hifn_process+0x1152/frame 0xc3ff1818
crypto_invoke(0,0,0,0,c5840bf4,...) at crypto_invoke+0x79/frame 0xc3ff184c
crypto_dispatch(c5840bf4,c1463a3c,368,c3ff18d3,0,...) at crypto_dispatch+0x64/frame 0xc3ff1870
esp_output(c48d1700,c475b380,0,14,9,...) at esp_output+0xb04/frame 0xc3ff1928
ipsec4_process_packet(c48d1700,c475b380,c3ff1ad8,0,c4ff6c80,...) at ipsec4_process_packet+0x2bf/frame 0xc3ff1a1c
ip_ipsec_output(c3ff1adc,0,c3ff1ad8,1,10000000,...) at ip_ipsec_output+0xe1/frame 0xc3ff1a40
ip_output(c48d1700,0,c3ff1b48,1,0,...) at ip_output+0x9c7/frame 0xc3ff1b00
ip_forward(c48d1700,0,0,1,0,...) at ip_forward+0x3ea/frame 0xc3ff1b88
ip_input(c48d1700,c4278c80,c3ff1c68,18c83,2710,...) at ip_input+0xba8/frame 0xc3ff1bf0
swi_net(c2427880,0,246,0,49990c6f,...) at swi_net+0x15f/frame 0xc3ff1c3c
intr_event_execute_handlers(109,c427b400,c1433a6b,55b,0,...) at intr_event_execute_handlers+0xaa/frame 0xc3ff1c68
ithread_loop(c41b4150,c3ff1ce8,0,0,0,...) at ithread_loop+0x80/frame 0xc3ff1ca4
fork_exit(c0ca75c0,c41b4150,c3ff1ce8) at fork_exit+0xa3/frame 0xc3ff1cd4
fork_trampoline() at fork_trampoline+0x8/frame 0xc3ff1cd4
--- trap 0, eip = 0, esp = 0xc3ff1d20, ebp = 0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
93354 47876 21 0 S nanslp 0xc1e95870 sleep
86366 85907 86366 0 S+ ttyin 0xc4309470 sh
85907 85197 85907 0 S+ wait 0xc5673308 sh
85218 55861 85218 0 Ss (threaded) sshlockout_pf
100136 S nanslp 0xc1e95870 sshlockout_pf
100118 S piperd 0xc478b198 sshlockout_pf
85197 1 85197 0 Ss+ wait 0xc486a918 login
85085 1 1 0 S nanslp 0xc1e95870 getty
65656 1 65656 0 Ss (threaded) apcupsd
100129 S accept 0xc4885536 apcupsd
100128 S sigwait 0xc5797000 apcupsd
100109 S select 0xc49b3124 apcupsd
65157 64773 64773 0 S nanslp 0xc1e95870 minicron
64773 1 64773 0 Ss wait 0xc4869000 minicron
64552 64470 64470 0 S nanslp 0xc1e95870 minicron
64470 1 64470 0 Ss wait 0xc4c80918 minicron
64269 63596 63596 0 S nanslp 0xc1e95870 minicron
63596 1 63596 0 Ss wait 0xc5673c20 minicron
58762 1 58762 65534 Ss select 0xc43be4a4 softflowd
55861 1 55861 0 Ss select 0xc49b60e4 syslogd
50520 1 50520 0 Ss select 0xc49b3de4 miniupnpd
47876 1 21 0 S+ wait 0xc4c80000 sh
40483 1 40483 0 Ss select 0xc49b4024 bsnmpd
39862 39654 39862 0 Ss (threaded) charon
100106 S uwait 0xc4c62680 charon
100105 S uwait 0xc4c62600 charon
100104 S uwait 0xc4ff6900 charon
100103 S uwait 0xc4ff6100 charon
100102 S uwait 0xc4ff6380 charon
100101 S uwait 0xc4ff6400 charon
100100 S uwait 0xc4ff6180 charon
100099 S uwait 0xc4758400 charon
100098 S uwait 0xc4ff6480 charon
100097 S uwait 0xc4ff6500 charon
100096 S uwait 0xc4ff6580 charon
100095 S uwait 0xc4734300 charon
100094 S uwait 0xc498f380 charon
100093 S select 0xc49b4224 charon
100092 S select 0xc49b4264 charon
100091 S uwait 0xc4c62c00 charon
100090 S sigwait 0xc5095000 charon
39654 1 39654 0 Ss select 0xc49b41e4 starter
36482 1 36482 0 Ss kqread 0xc4ff7400 dhcpleases
35474 1 35474 1002 Ss select 0xc49b48a4 dhcpd
30851 1 30851 0 Ss (threaded) ntpd
100087 S usem 0xc4c62a00 ntpd
100086 S select 0xc49b4824 ntpd
30222 1 30222 0 Ss nanslp 0xc1e95870 cron
29849 29259 29259 0 S kqread 0xc4c61e80 nginx
29598 29259 29259 0 S kqread 0xc4c61e00 nginx
29259 1 29259 0 Ss pause 0xc4c7f97c nginx
21349 1 21047 65534 S select 0xc49b4aa4 dnsmasq
19216 1 19216 0 Rs (threaded) dpinger
100080 S accept 0xc4ab0536 dpinger
100079 S nanslp 0xc1e95870 dpinger
100078 RunQ dpinger
100077 S sbwait 0xc4ab0408 dpinger
100076 S uwait 0xc4734880 dpinger
15422 1 15422 0 Rs filterlog
14645 1 14645 0 Ss select 0xc49b52a4 openvpn
12009 1 12009 65 Ss select 0xc49b5fa4 dhclient
6466 1 6466 0 Ss select 0xc49b5ea4 dhclient
396 1 396 0 Ss select 0xc47188a4 devd
375 372 372 0 S kqread 0xc498ef00 check_reload_status
372 1 372 0 Ss kqread 0xc498e500 check_reload_status
317 1 317 0 Rs php-fpm
52 0 0 0 DL mdwait 0xc471b800 [md1]
47 0 0 0 DL mdwait 0xc4715000 [md0]
20 0 0 0 DL syncer 0xc1eb4bc4 [syncer]
19 0 0 0 DL vlruwt 0xc473a000 [vnlru]
18 0 0 0 DL psleep 0xc1eb4304 [bufdaemon]
17 0 0 0 DL pollid 0xc1e943e0 [idlepoll]
9 0 0 0 DL pgzero 0xc1ebeda8 [pagezero]
8 0 0 0 DL psleep 0xc1ebeac4 [vmdaemon]
7 0 0 0 DL (threaded) [pagedaemon]
100053 D umarcl 0xc1ebe6a4 [uma]
100046 D psleep 0xc1f1e584 [pagedaemon]
6 0 0 0 DL waiting_ 0xc1f18ef4 [sctp_iterator]
5 0 0 0 RL [pf purge]
16 0 0 0 DL (threaded) [usb]
100038 D - 0xc439fd64 [usbus1]
100037 D - 0xc439fd34 [usbus1]
100036 D - 0xc439fd04 [usbus1]
100035 D - 0xc439fcd4 [usbus1]
100034 D - 0xc439fca4 [usbus1]
100033 D - 0xc43a8b8c [usbus0]
100032 D - 0xc43a8b5c [usbus0]
100031 D - 0xc43a8b2c [usbus0]
100030 D - 0xc43a8afc [usbus0]
100029 D - 0xc43a8acc [usbus0]
4 0 0 0 DL (threaded) [cam]
100045 D - 0xc1e676ac [scanner]
100015 D - 0xc1e67800 [doneq0]
15 0 0 0 RL [rand_harvestq]
3 0 0 0 DL crypto_r 0xc1ebda28 [crypto returns]
2 0 0 0 DL crypto_w 0xc1ebd968 [crypto]
14 0 0 0 DL (threaded) [geom]
100010 D - 0xc1f13f94 [g_down]
100009 D - 0xc1f13f90 [g_up]
100008 D - 0xc1f13f8c [g_event]
13 0 0 0 DL sleep 0xc1e3f458 [ng_queue0]
12 0 0 0 RL (threaded) [intr]
100043 I [swi1: pfsync]
100041 I [swi1: pf send]
100039 I [swi0: uart uart]
100028 I [irq12: ohci0 ehci0]
100027 I [irq15: vr2 ata1]
100026 I [irq14: ata0]
100025 I [irq9: hifn0]
100024 RunQ [swi5: fast taskq]
100022 I [swi6: Giant taskq]
100020 I [swi6: task queue]
100006 RunQ [swi4: clock]
100005 I [swi3: vm]
100004 Run CPU 0 [swi1: netisr 0]
11 0 0 0 RL [idle: cpu0]
1 0 1 0 SLs wait 0xc4276308 [init]
10 0 0 0 DL audit_wo 0xc1f1ce5c [audit]
0 0 0 0 DLs (threaded) [kernel]
100044 D - 0xc4279a00 [CAM taskq]
100023 D - 0xc4279400 [thread taskq]
100021 D - 0xc4279580 [ffs_trim taskq]
100019 D - 0xc4279700 [acpi_task_2]
100018 D - 0xc4279700 [acpi_task_1]
100017 D - 0xc4279700 [acpi_task_0]
100016 D - 0xc4279780 [kqueue taskq]
100011 D - 0xc427ad00 [firmware taskq]
100000 D swapin 0xc1f14014 [swapper]
db:0:kdb.enter.default> alltrace

Tracing command charon pid 39862 tid 100106 td 0xc4c85960
sched_switch(c4c85960,0,104,d3354a28,0,...) at sched_switch+0x2d2/frame 0xd3354a08
mi_switch(104,0,d3354ab8,c0cc34e2,c1ea1a80,c4c85960) at mi_switch+0x122/frame 0xd3354a40
sleepq_switch(c4c85960,0,c143d24a,1a8,c4c85960,...) at sleepq_switch+0x15b/frame 0xd3354a68
sleepq_catch_signals(0,c143d24a,154,200007f,1fffff,...) at sleepq_catch_signals+0x5be/frame 0xd3354abc
sleepq_wait_sig(c4c62680,0,c1439ec2,100,0,...) at sleepq_wait_sig+0x14/frame 0xd3354ae8
sleep(c4c62680,c1ea0994,300,c1439ec2,0,...) at _sleep+0x29b/frame 0xd3354b30
umtxq_sleep(0,d3354bb8,0,c4c62688,d3354c18,...) at umtxq_sleep+0x125/frame 0xd3354b90
do_wait(0,0,1,1,14,...) at do_wait+0x371/frame 0xd3354be8
_umtx_op_wait_uint_private(c4c85960,d3354ca8,c0d0c982,c4c85960,c4c83320,...) at _umtx_op_wait_uint_private+0x9f/frame 0xd3354c18
syscall(d3354ce8) at syscall+0x5c9/frame 0xd3354cdc
Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xd3354cdc
--- syscall (454, FreeBSD ELF32, sys_umtx_op), eip = 0x281781a3, esp = 0xbaceedf4, ebp = 0xbaceee38 ---

Tracing command charon pid 39862 tid 100105 td 0xc4c85c80
sched_switch(c4c85c80,0,104,b0,98,...) at sched_switch+0x2d2/frame 0xd3351a08
mi_switch(104,0,d3351ab8,c0cc34e2,c1ea1a80,c4c85c80) at mi_switch+0x122/frame 0xd3351a40
sleepq_switch(c4c85c80,0,c143d24a,1a8,c4ab16f4,...) at sleepq_switch+0x15b/frame 0xd3351a68
sleepq_catch_signals(0,c143d24a,154,c4ab16a0,d3351b10,...) at sleepq_catch_signals+0x5be/frame 0xd3351abc
sleepq_wait_sig(c4c62600,0,c1439ec2,100,0,...) at sleepq_wait_sig+0x14/frame 0xd3351ae8
sleep(c4c62600,c1e9e6e4,300,c1439ec2,0,...) at _sleep+0x29b/frame 0xd3351b30
umtxq_sleep(0,d3351bb8,0,c4c62608,c4c85c80,...) at umtxq_sleep+0x125/frame 0xd3351b90
do_wait(0,0,1,1,14,...) at do_wait+0x371/frame 0xd3351be8
_umtx_op_wait_uint_private(c4c85c80,d3351ca8,c0d0c982,c4c85c80,c4c83c80,...) at _umtx_op_wait_uint_private+0x9f/frame 0xd3351c18
syscall(d3351ce8) at syscall+0x5c9/frame 0xd3351cdc
Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xd3351cdc
--- syscall (454, FreeBSD ELF32, sys_umtx_op), eip = 0x281781a3, esp = 0xbadefdf4, ebp = 0xbadefe38 ---

Tracing command charon pid 39862 tid 100104 td 0xc50ae000
sched_switch(c50ae000,0,104,0,0,...) at sched_switch+0x2d2/frame 0xd334ea08
mi_switch(104,0,d334eab8,c0cc34e2,c1ea1a80,c50ae000) at mi_switch+0x122/frame 0xd334ea40
sleepq_switch(c50ae000,0,c143d24a,1a8,100,...) at sleepq_switch+0x15b/frame 0xd334ea68
sleepq_catch_signals(0,c143d24a,154,100,0,...) at sleepq_catch_signals+0x5be/frame 0xd334eabc
sleepq_wait_sig(c4ff6900,0,c1439ec2,100,0,...) at sleepq_wait_sig+0x14/frame 0xd334eae8
sleep(c4ff6900,c1e9c464,300,c1439ec2,0,...) at _sleep+0x29b/frame 0xd334eb30
umtxq_sleep(0,d334ebb8,0,c4ff6908,d334ebf8,...) at umtxq_sleep+0x125/frame 0xd334eb90
do_wait(0,0,1,1,14,...) at do_wait+0x371/frame 0xd334ebe8
_umtx_op_wait_uint_private(c50ae000,d334eca8,c0d0c982,c50ae000,c4c83c80,...) at _umtx_op_wait_uint_private+0x9f/frame 0xd334ec18
syscall(d334ece8) at syscall+0x5c9/frame 0xd334ecdc
Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xd334ecdc
--- syscall (454, FreeBSD ELF32, sys_umtx_op), eip = 0x281781a3, esp = 0xbaef0df4, ebp = 0xbaef0e38 ---

------------ output truncated --------------

Norm Sevilla wrote:

I can confirm the issue on my setup as well, a Negate m1n1wall with pfSense 2.3.3_1 and a Hifn 7955 crypto accelerator card. As soon as traffic is initiated to hosts defined in IPSec Phase 2 entries, Fatal trap 12 occurs and then a lot of trace logs from charon process. Crash only happens with md5 3DES used for cipher. When using SHA AES, device doesn't crash, but it doesn't pass any encrpyted traffic either. Phase 1 is established but no traffic through Phase 2.

When crypto card is removed, SHA AES and md5 3DES works fine.

Tracing pid 12 tid 100004 td 0xc4278c80
bcopy(0,d31f9738,18,c07f62e0,c4bcea78,...) at bcopy+0x1a/frame 0xc3ff1798
hifn_process(c41c2a80,c5840bf4,0,c27e0820,0,...) at hifn_process+0x1152/frame 0xc3ff1818
crypto_invoke(0,0,0,0,c5840bf4,...) at crypto_invoke+0x79/frame 0xc3ff184c
crypto_dispatch(c5840bf4,c1463a3c,368,c3ff18d3,0,...) at crypto_dispatch+0x64/frame 0xc3ff1870
esp_output(c48d1700,c475b380,0,14,9,...) at esp_output+0xb04/frame 0xc3ff1928
ipsec4_process_packet(c48d1700,c475b380,c3ff1ad8,0,c4ff6c80,...) at ipsec4_process_packet+0x2bf/frame 0xc3ff1a1c
ip_ipsec_output(c3ff1adc,0,c3ff1ad8,1,10000000,...) at ip_ipsec_output+0xe1/frame 0xc3ff1a40
ip_output(c48d1700,0,c3ff1b48,1,0,...) at ip_output+0x9c7/frame 0xc3ff1b00
ip_forward(c48d1700,0,0,1,0,...) at ip_forward+0x3ea/frame 0xc3ff1b88
ip_input(c48d1700,c4278c80,c3ff1c68,18c83,2710,...) at ip_input+0xba8/frame 0xc3ff1bf0
swi_net(c2427880,0,246,0,49990c6f,...) at swi_net+0x15f/frame 0xc3ff1c3c
intr_event_execute_handlers(109,c427b400,c1433a6b,55b,0,...) at intr_event_execute_handlers+0xaa/frame 0xc3ff1c68
ithread_loop(c41b4150,c3ff1ce8,0,0,0,...) at ithread_loop+0x80/frame 0xc3ff1ca4
fork_exit(c0ca75c0,c41b4150,c3ff1ce8) at fork_exit+0xa3/frame 0xc3ff1cd4
fork_trampoline() at fork_trampoline+0x8/frame 0xc3ff1cd4
--- trap 0, eip = 0, esp = 0xc3ff1d20, ebp = 0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
93354 47876 21 0 S nanslp 0xc1e95870 sleep
86366 85907 86366 0 S+ ttyin 0xc4309470 sh
85907 85197 85907 0 S+ wait 0xc5673308 sh
85218 55861 85218 0 Ss (threaded) sshlockout_pf
100136 S nanslp 0xc1e95870 sshlockout_pf
100118 S piperd 0xc478b198 sshlockout_pf
85197 1 85197 0 Ss+ wait 0xc486a918 login
85085 1 1 0 S nanslp 0xc1e95870 getty
65656 1 65656 0 Ss (threaded) apcupsd
100129 S accept 0xc4885536 apcupsd
100128 S sigwait 0xc5797000 apcupsd
100109 S select 0xc49b3124 apcupsd
65157 64773 64773 0 S nanslp 0xc1e95870 minicron
64773 1 64773 0 Ss wait 0xc4869000 minicron
64552 64470 64470 0 S nanslp 0xc1e95870 minicron
64470 1 64470 0 Ss wait 0xc4c80918 minicron
64269 63596 63596 0 S nanslp 0xc1e95870 minicron
63596 1 63596 0 Ss wait 0xc5673c20 minicron
58762 1 58762 65534 Ss select 0xc43be4a4 softflowd
55861 1 55861 0 Ss select 0xc49b60e4 syslogd
50520 1 50520 0 Ss select 0xc49b3de4 miniupnpd
47876 1 21 0 S+ wait 0xc4c80000 sh
40483 1 40483 0 Ss select 0xc49b4024 bsnmpd
39862 39654 39862 0 Ss (threaded) charon
100106 S uwait 0xc4c62680 charon
100105 S uwait 0xc4c62600 charon
100104 S uwait 0xc4ff6900 charon
100103 S uwait 0xc4ff6100 charon
100102 S uwait 0xc4ff6380 charon
100101 S uwait 0xc4ff6400 charon
100100 S uwait 0xc4ff6180 charon
100099 S uwait 0xc4758400 charon
100098 S uwait 0xc4ff6480 charon
100097 S uwait 0xc4ff6500 charon
100096 S uwait 0xc4ff6580 charon
100095 S uwait 0xc4734300 charon
100094 S uwait 0xc498f380 charon
100093 S select 0xc49b4224 charon
100092 S select 0xc49b4264 charon
100091 S uwait 0xc4c62c00 charon
100090 S sigwait 0xc5095000 charon
39654 1 39654 0 Ss select 0xc49b41e4 starter
36482 1 36482 0 Ss kqread 0xc4ff7400 dhcpleases
35474 1 35474 1002 Ss select 0xc49b48a4 dhcpd
30851 1 30851 0 Ss (threaded) ntpd
100087 S usem 0xc4c62a00 ntpd
100086 S select 0xc49b4824 ntpd
30222 1 30222 0 Ss nanslp 0xc1e95870 cron
29849 29259 29259 0 S kqread 0xc4c61e80 nginx
29598 29259 29259 0 S kqread 0xc4c61e00 nginx
29259 1 29259 0 Ss pause 0xc4c7f97c nginx
21349 1 21047 65534 S select 0xc49b4aa4 dnsmasq
19216 1 19216 0 Rs (threaded) dpinger
100080 S accept 0xc4ab0536 dpinger
100079 S nanslp 0xc1e95870 dpinger
100078 RunQ dpinger
100077 S sbwait 0xc4ab0408 dpinger
100076 S uwait 0xc4734880 dpinger
15422 1 15422 0 Rs filterlog
14645 1 14645 0 Ss select 0xc49b52a4 openvpn
12009 1 12009 65 Ss select 0xc49b5fa4 dhclient
6466 1 6466 0 Ss select 0xc49b5ea4 dhclient
396 1 396 0 Ss select 0xc47188a4 devd
375 372 372 0 S kqread 0xc498ef00 check_reload_status
372 1 372 0 Ss kqread 0xc498e500 check_reload_status
317 1 317 0 Rs php-fpm
52 0 0 0 DL mdwait 0xc471b800 [md1]
47 0 0 0 DL mdwait 0xc4715000 [md0]
20 0 0 0 DL syncer 0xc1eb4bc4 [syncer]
19 0 0 0 DL vlruwt 0xc473a000 [vnlru]
18 0 0 0 DL psleep 0xc1eb4304 [bufdaemon]
17 0 0 0 DL pollid 0xc1e943e0 [idlepoll]
9 0 0 0 DL pgzero 0xc1ebeda8 [pagezero]
8 0 0 0 DL psleep 0xc1ebeac4 [vmdaemon]
7 0 0 0 DL (threaded) [pagedaemon]
100053 D umarcl 0xc1ebe6a4 [uma]
100046 D psleep 0xc1f1e584 [pagedaemon]
6 0 0 0 DL waiting_ 0xc1f18ef4 [sctp_iterator]
5 0 0 0 RL [pf purge]
16 0 0 0 DL (threaded) [usb]
100038 D - 0xc439fd64 [usbus1]
100037 D - 0xc439fd34 [usbus1]
100036 D - 0xc439fd04 [usbus1]
100035 D - 0xc439fcd4 [usbus1]
100034 D - 0xc439fca4 [usbus1]
100033 D - 0xc43a8b8c [usbus0]
100032 D - 0xc43a8b5c [usbus0]
100031 D - 0xc43a8b2c [usbus0]
100030 D - 0xc43a8afc [usbus0]
100029 D - 0xc43a8acc [usbus0]
4 0 0 0 DL (threaded) [cam]
100045 D - 0xc1e676ac [scanner]
100015 D - 0xc1e67800 [doneq0]
15 0 0 0 RL [rand_harvestq]
3 0 0 0 DL crypto_r 0xc1ebda28 [crypto returns]
2 0 0 0 DL crypto_w 0xc1ebd968 [crypto]
14 0 0 0 DL (threaded) [geom]
100010 D - 0xc1f13f94 [g_down]
100009 D - 0xc1f13f90 [g_up]
100008 D - 0xc1f13f8c [g_event]
13 0 0 0 DL sleep 0xc1e3f458 [ng_queue0]
12 0 0 0 RL (threaded) [intr]
100043 I [swi1: pfsync]
100041 I [swi1: pf send]
100039 I [swi0: uart uart]
100028 I [irq12: ohci0 ehci0]
100027 I [irq15: vr2 ata1]
100026 I [irq14: ata0]
100025 I [irq9: hifn0]
100024 RunQ [swi5: fast taskq]
100022 I [swi6: Giant taskq]
100020 I [swi6: task queue]
100006 RunQ [swi4: clock]
100005 I [swi3: vm]
100004 Run CPU 0 [swi1: netisr 0]
11 0 0 0 RL [idle: cpu0]
1 0 1 0 SLs wait 0xc4276308 [init]
10 0 0 0 DL audit_wo 0xc1f1ce5c [audit]
0 0 0 0 DLs (threaded) [kernel]
100044 D - 0xc4279a00 [CAM taskq]
100023 D - 0xc4279400 [thread taskq]
100021 D - 0xc4279580 [ffs_trim taskq]
100019 D - 0xc4279700 [acpi_task_2]
100018 D - 0xc4279700 [acpi_task_1]
100017 D - 0xc4279700 [acpi_task_0]
100016 D - 0xc4279780 [kqueue taskq]
100011 D - 0xc427ad00 [firmware taskq]
100000 D swapin 0xc1f14014 [swapper]
db:0:kdb.enter.default> alltrace

Tracing command charon pid 39862 tid 100106 td 0xc4c85960
sched_switch(c4c85960,0,104,d3354a28,0,...) at sched_switch+0x2d2/frame 0xd3354a08
mi_switch(104,0,d3354ab8,c0cc34e2,c1ea1a80,c4c85960) at mi_switch+0x122/frame 0xd3354a40
sleepq_switch(c4c85960,0,c143d24a,1a8,c4c85960,...) at sleepq_switch+0x15b/frame 0xd3354a68
sleepq_catch_signals(0,c143d24a,154,200007f,1fffff,...) at sleepq_catch_signals+0x5be/frame 0xd3354abc
sleepq_wait_sig(c4c62680,0,c1439ec2,100,0,...) at sleepq_wait_sig+0x14/frame 0xd3354ae8
sleep(c4c62680,c1ea0994,300,c1439ec2,0,...) at _sleep+0x29b/frame 0xd3354b30
umtxq_sleep(0,d3354bb8,0,c4c62688,d3354c18,...) at umtxq_sleep+0x125/frame 0xd3354b90
do_wait(0,0,1,1,14,...) at do_wait+0x371/frame 0xd3354be8
_umtx_op_wait_uint_private(c4c85960,d3354ca8,c0d0c982,c4c85960,c4c83320,...) at _umtx_op_wait_uint_private+0x9f/frame 0xd3354c18
syscall(d3354ce8) at syscall+0x5c9/frame 0xd3354cdc
Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xd3354cdc
--- syscall (454, FreeBSD ELF32, sys_umtx_op), eip = 0x281781a3, esp = 0xbaceedf4, ebp = 0xbaceee38 ---

Tracing command charon pid 39862 tid 100105 td 0xc4c85c80
sched_switch(c4c85c80,0,104,b0,98,...) at sched_switch+0x2d2/frame 0xd3351a08
mi_switch(104,0,d3351ab8,c0cc34e2,c1ea1a80,c4c85c80) at mi_switch+0x122/frame 0xd3351a40
sleepq_switch(c4c85c80,0,c143d24a,1a8,c4ab16f4,...) at sleepq_switch+0x15b/frame 0xd3351a68
sleepq_catch_signals(0,c143d24a,154,c4ab16a0,d3351b10,...) at sleepq_catch_signals+0x5be/frame 0xd3351abc
sleepq_wait_sig(c4c62600,0,c1439ec2,100,0,...) at sleepq_wait_sig+0x14/frame 0xd3351ae8
sleep(c4c62600,c1e9e6e4,300,c1439ec2,0,...) at _sleep+0x29b/frame 0xd3351b30
umtxq_sleep(0,d3351bb8,0,c4c62608,c4c85c80,...) at umtxq_sleep+0x125/frame 0xd3351b90
do_wait(0,0,1,1,14,...) at do_wait+0x371/frame 0xd3351be8
_umtx_op_wait_uint_private(c4c85c80,d3351ca8,c0d0c982,c4c85c80,c4c83c80,...) at _umtx_op_wait_uint_private+0x9f/frame 0xd3351c18
syscall(d3351ce8) at syscall+0x5c9/frame 0xd3351cdc
Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xd3351cdc
--- syscall (454, FreeBSD ELF32, sys_umtx_op), eip = 0x281781a3, esp = 0xbadefdf4, ebp = 0xbadefe38 ---

Tracing command charon pid 39862 tid 100104 td 0xc50ae000
sched_switch(c50ae000,0,104,0,0,...) at sched_switch+0x2d2/frame 0xd334ea08
mi_switch(104,0,d334eab8,c0cc34e2,c1ea1a80,c50ae000) at mi_switch+0x122/frame 0xd334ea40
sleepq_switch(c50ae000,0,c143d24a,1a8,100,...) at sleepq_switch+0x15b/frame 0xd334ea68
sleepq_catch_signals(0,c143d24a,154,100,0,...) at sleepq_catch_signals+0x5be/frame 0xd334eabc
sleepq_wait_sig(c4ff6900,0,c1439ec2,100,0,...) at sleepq_wait_sig+0x14/frame 0xd334eae8
sleep(c4ff6900,c1e9c464,300,c1439ec2,0,...) at _sleep+0x29b/frame 0xd334eb30
umtxq_sleep(0,d334ebb8,0,c4ff6908,d334ebf8,...) at umtxq_sleep+0x125/frame 0xd334eb90
do_wait(0,0,1,1,14,...) at do_wait+0x371/frame 0xd334ebe8
_umtx_op_wait_uint_private(c50ae000,d334eca8,c0d0c982,c50ae000,c4c83c80,...) at _umtx_op_wait_uint_private+0x9f/frame 0xd334ec18
syscall(d334ece8) at syscall+0x5c9/frame 0xd334ecdc
Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xd334ecdc
--- syscall (454, FreeBSD ELF32, sys_umtx_op), eip = 0x281781a3, esp = 0xbaef0df4, ebp = 0xbaef0e38 ---

------------ output truncated --------------