Project

General

Profile

Bug #9123

Adding/configuring vlan on ixl-devices causes aq_add_macvlan err -53, aq_error 14

Added by Sebastian Deuerling almost 2 years ago. Updated 3 months ago.

Status:
New
Priority:
Very High
Assignee:
Category:
Interfaces
Target version:
-
Start date:
11/15/2018
Due date:
% Done:

0%

Estimated time:
Affected Version:
2.4.x
Affected Architecture:
amd64

Description

The actual vlan addition/configuring process is triggering error "aq_add_macvlan err -53, aq_error 14" on ixl-devices.
Configuring vlans seems to work nevertheless, but saving interface configurations with vlans takes a lot of time.
In our setup (two igb-interfaces, two ix-interfaces, two ixl-interfaces; 25 vlans on failover-lagg of ixl0 and igb0) saving changes on interface configuration lasts around about 20 to 30 minutes. After that pfSense seems to freeze. After reboot all vlans are working.
But booting also takes a lof of time. Around 5 minutes in step "Configuring VLANS...".
Our hardware: SYS-5018D-FN4T (Supermicro Intel Xeon D-1541 system) and X710DA2BLK (Intel X710-DA2 Dual-SFP+-PCIe-Addon-cards).
Further information here: https://forum.netgate.com/topic/136201/new-version-2-4-4-interface-error-aq_add_macvlan-err-53-aq_error-14/14

History

#1 Updated by Steve Wheeler over 1 year ago

  • Category set to Interfaces
  • Affected Version set to 2.4.x
  • Affected Architecture amd64 added
  • Affected Architecture deleted ()

#2 Updated by Alex Rosenberg over 1 year ago

Not sure about the similarity of conditions yet, but I'm seeing this message being logged on my FreeNAS box with the ixl driver on top of the SuperMicro X11DPH-Tq motherboard. I'm beginning to suspect a problem either with the ixl driver or the NVM fw image.

#3 Updated by Alexander Meckelein over 1 year ago

Same problem here.

Hardware: Dell PowerEdge 330 with Intel(R) 10GbE 2P X710 Adapter

Done so far:
  • installation of PFSense 2.4.4 with "pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64" Image, used option ZFS mirror on 2 SSDs
  • on first boot of installed PFSense I configured the "ixl0" interface with VLAN Tag, immediately got the error "aq_add_macvlan err -53, aq_error 14" in the terminal.
  • reset installation to factory default, reboot
  • tested same on the other interface "ixl1", got the same error
  • both interfaces were not able to get IP by DHCP

Atm I did not test the PFSense 2.4.3 image on this system, this is planned for this week on friday.

#4 Updated by Alexander Meckelein over 1 year ago

I have tested the FreeBSD Version 11.1, 11.2 and 12.0 on the Hardware and got following results.

FreeBSD 11.1 (FreeBSD-11.1-RELEASE-amd64-memstick) (Driver 1.7.12-k)
ifconfig ixl0.1 create vlan 1 vlandev ixl0 inet 192.168.23.223/24
ifconfig ixl0 up

-> Ping OK

FreeBSD 11.2 (FreeBSD-11.2-RELEASE-amd64-memstick) (Driver 1.9.9-k)
ifconfig ixl0.1 create vlan 1 vlandev ixl0 inet 192.168.23.223/24
terminal message: ixl0: aq_add_macvlan err -53, aq_error 14
ifconfig ixl0 up
terminal message: ixl0: aq_add_macvlan err -53, aq_error 14
-> Ping OK (448 Packets, 0% lost)

FreeBSD 12 (FreeBSD-12.0-RELEASE-amd64-memstick)
ifconfig ixl0.1 create vlan 1 vlandev ixl0 inet 192.168.23.223/24
ifconfig ixl0 up

-> Ping OK

#5 Updated by Eric Machabert 10 months ago

Hi,

As I explained in the forum : this is my currently working solution while runing 2.4.4p3:
- Using lagg in failover mode and not LACP (using LACP requires to disable firmware based LLDP agent per Intel documentation. we did not test it)
- Using 1.11.20 driver (september 2019) from Intel website

We do use lots of vlan and we don't see any issue after 15 days of uptime. We did test CARP failover, interface assignment on newly declared vlan and had no "Queue appears to be hung" issue.

information regarding the NICs (HPE NC562SFP+):

ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.11.20> mem
0xe4000000-0xe4ffffff,0xe5008000-0xe500ffff at device 0.0 numa-domain 0 on pci4
ixl0: using 1024 tx descriptors and 1024 rx descriptors
ixl0: fw 6.71.49427 api 1.7 nvm 6.80 etid 80004004 oem 1.263.0

#6 Updated by Luiz Souza 5 months ago

  • Assignee set to Luiz Souza
  • Priority changed from Normal to Very High

#7 Updated by Marc L 3 months ago

Some more data/observations:
  • NIC: Intel X710-DA4 (Quad Port 10Gb)
  • pfSense version 2.4.4-p3
  • One LAGG group (lagg0) Protocol set to LACP
  • 31 VLANs (all lagg0.x)
  • Adding more VLANs on-the-fly works for us, though we do it rarely. We used to have the LAGG Protocol set to "NONE" which would consistently produce crashes when adding a new VLAN
  • Applying Interface settings is pretty slow but works
  • However i did encounter a crash when changing the description of a VLAN (on the "VLANs" tab)

The logs collected from the crash make it look like changing the description and saving caused all VLANs to be taken down and reconfigured. Shouldn't be necessary in the first place, because the description is just webinterface cosmetics, buts that's a different topic.

I followed this procedure twice (edit VLAN -> set new description -> hit save). Was physically present and watched the console. The first time around, the webinterface hung for a while, and the mentioned error was logged to the console a few times, but it succeeded eventually. The second time the webinterface hung again, saw a few of the error messages on the console and ultimately the system crashed and rebooted. Below is the relevant tail end of the collected logs from the crash report.

msgbuf.txt:

ixl0: aq_add_macvlan err -53, aq_error 14
<6>carp: 9@lagg0.101: BACKUP -> INIT (hardware interface down)
<6>carp: demoted by 240 to 240 (interface down)
<7>ifa_maintain_loopback_route: deletion failed for interface lagg0.101: 3
<7>ifa_maintain_loopback_route: deletion failed for interface lagg0.101: 3
<7>ifa_maintain_loopback_route: deletion failed for interface lagg0.101: 3
<6>carp: demoted by -240 to 0 (vhid removed)
<6>lagg0.101: promiscuous mode disabled
<6>vlan5: changing name to 'lagg0.101'
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14
<6>lagg0.101: promiscuous mode enabled
<6>carp: 9@lagg0.101: INIT -> BACKUP (initialization complete)
<6>carp: 19@lagg0.205: BACKUP -> MASTER (master timed out)
<6>carp: 10@lagg0.102: BACKUP -> MASTER (master timed out)
<6>carp: 13@lagg0.112: BACKUP -> MASTER (master timed out)
<6>carp: 34@lagg0.82: BACKUP -> MASTER (master timed out)
<6>carp: 8@lagg0.100: BACKUP -> MASTER (master timed out)

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer    = 0x20:0xffffffff80e38d40
stack pointer            = 0x28:0xfffffe04549b48b0
frame pointer            = 0x28:0xfffffe04549b48f0
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 12 (swi4: clock (0))

version.txt

FreeBSD 11.2-RELEASE-p10 #9 4a2bfdce133(RELENG_2_4_4): Wed May 15 18:54:42 EDT 2019
    root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense

ddb.txt

db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x898380
curthread    = 0xfffff80008438620: pid 12 "swi4: clock (0)" 
curpcb       = 0xfffffe04549b4b80
fpcurthread  = none
idlethread   = 0xfffff800083d5000: tid 100003 "idle: cpu0" 
curpmap      = 0xffffffff82b85998
tssp         = 0xffffffff82bb6810
commontssp   = 0xffffffff82bb6810
rsp0         = 0xfffffe04549b4b80
gs32p        = 0xffffffff82bbd068
ldt          = 0xffffffff82bbd0a8
tss          = 0xffffffff82bbd098
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100026 td 0xfffff80008438620
carp_master_down_locked() at carp_master_down_locked+0xf0/frame 0xfffffe04549b48f0
carp_master_down() at carp_master_down+0x21/frame 0xfffffe04549b4910
softclock_call_cc() at softclock_call_cc+0x13a/frame 0xfffffe04549b49c0
softclock() at softclock+0x79/frame 0xfffffe04549b49e0
intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe04549b4a20
ithread_loop() at ithread_loop+0xe7/frame 0xfffffe04549b4a70
fork_exit() at fork_exit+0x83/frame 0xfffffe04549b4ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe04549b4ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
86643   453   453     0  R       CPU 3                       devd
 8390  8805   349     0  S       select   0xfffff8019b937240 ifconfig
98627 90997    31     0  S       nanslp   0xffffffff829d1701 sleep
13432 95873   349     0  S       nanslp   0xffffffff829d1700 sleep
  138     1   138   136  Ss      select   0xfffff8019b775bc0 dhcpd
52965 51991 52965     0  S+      ttyin    0xfffff8001f27b0b0 sh
52789 52507 51844     0  S       piperd   0xfffff8001fd252f8 sh
52507 51844 51844     0  S       wait     0xfffff8019ba1b000 sh
52416 51844 51844     0  S       (threaded)                  sshg-blocker
100169                   S       piperd   0xfffff8001fa2c5f0 sshg-blocker
100280                   S       nanslp   0xffffffff829d1700 sshg-blocker
52283 51844 51844     0  S       piperd   0xfffff8001f81d2f8 sshg-parser
52258 51844 51844     0  S       piperd   0xfffff8001f814000 cat
51991 50253 51991     0  S+      wait     0xfffff8001f1af590 sh
51844 61015 51844     0  Ss      wait     0xfffff8001f93c000 sh
51562     1 51562     0  Ss+     ttyin    0xfffff8001f2cb8b0 getty
51297     1 51297     0  Ss+     ttyin    0xfffff8001f2cbcb0 getty
51195     1 51195     0  Ss+     ttyin    0xfffff8001f22d4b0 getty
50972     1 50972     0  Ss+     ttyin    0xfffff8001f22d8b0 getty
50832     1 50832     0  Ss+     ttyin    0xfffff8001f22dcb0 getty
50577     1 50577     0  Ss+     ttyin    0xfffff8001f26d8b0 getty
50360     1 50360     0  Ss+     ttyin    0xfffff8001f26d0b0 getty
50253     1 50253     0  Ss+     wait     0xfffff8019b032000 login
98172     1 98172     0  Ss      (threaded)                  filterdns
100256                   S       uwait    0xfffff80076145c80 filterdns
100257                   S       uwait    0xfffff8006ffcee80 filterdns
100258                   S       uwait    0xfffff80146ffb480 filterdns
100259                   S       uwait    0xfffff80146ffb680 filterdns
100260                   S       uwait    0xfffff80146ffb800 filterdns
100261                   S       uwait    0xfffff8001f8da000 filterdns
100262                   S       uwait    0xfffff80146a1c100 filterdns
100263                   S       uwait    0xfffff8001f907c80 filterdns
100264                   S       uwait    0xfffff8006ffcd880 (REDACTED)
100265                   S       uwait    0xfffff8007625e600 (REDACTED)
100266                   S       uwait    0xfffff8007625e480 (REDACTED)
100267                   S       uwait    0xfffff8007625e880 (REDACTED)
100268                   S       uwait    0xfffff8007625f180 (REDACTED)
100269                   S       uwait    0xfffff8007625ea00 (REDACTED)
100270                   S       uwait    0xfffff8014656ec80 (REDACTED)
100271                   S       uwait    0xfffff80121e7c180 (REDACTED)
100272                   S       uwait    0xfffff8006fff2400 signal-thread
95873     1   349     0  S       wait     0xfffff80121d9d000 sh
92488 91912 91912   100  S       kqread   0xfffff801463d5a00 squid
91912     1 91912   100  Ss      wait     0xfffff8001f3c7000 squid
66256 65706 65706     0  S       nanslp   0xffffffff829d1703 minicron
65706     1 65706     0  Ss      wait     0xfffff8001f3e0000 minicron
65143 64986 64986     0  S       nanslp   0xffffffff829d1702 minicron
64986     1 64986     0  Ss      wait     0xfffff8001fd61000 minicron
64924 64452 64452     0  S       nanslp   0xffffffff829d1700 minicron
64452     1 64452     0  Ss      wait     0xfffff80121da0590 minicron
61353     1    31     0  S+      nanslp   0xffffffff829d1703 php-cgi
61015     1 61015     0  Ss      select   0xfffff8019b001640 syslogd
90997     1    31     0  S+      wait     0xfffff8001f9be590 sh
63866     1 63866     0  Ss      select   0xfffff80146dc94c0 bsnmpd
61072 60982 61072     0  Rs      (threaded)                  charon
100184                   S       sigwait  0xfffff80076308000 charon
100225                   S       uwait    0xfffff8001f8dde00 charon
100226                   Run     CPU 2                       charon
100227                   S       select   0xfffff80146dc9a40 charon
100228                   S       uwait    0xfffff80121b5c780 charon
100229                   S       uwait    0xfffff80121aa8d00 charon
100230                   S       uwait    0xfffff801467d9800 charon
100231                   S       uwait    0xfffff801467d9a00 charon
100232                   S       uwait    0xfffff8007621f980 charon
100233                   S       uwait    0xfffff8007621fa80 charon
100234                   S       uwait    0xfffff80146a1c000 charon
100235                   S       uwait    0xfffff80121ea6c00 charon
100236                   S       uwait    0xfffff8001f912800 charon
100237                   S       uwait    0xfffff8007621f480 charon
100238                   S       uwait    0xfffff8007621f580 charon
100239                   S       uwait    0xfffff8007621f680 charon
100240                   S       uwait    0xfffff8007621f780 charon
60982     1 60982     0  Ss      select   0xfffff80146dc98c0 starter
73636     1 73636     0  Rs      (threaded)                  ntpd
100222                   Run     CPU 1                       ntpd
100305                   S       usem     0xfffff8019b778d80 ntpd
73011     1 73011     0  Ss      nanslp   0xffffffff829d1701 cron
72533 72077 72077     0  S       kqread   0xfffff80121814b00 nginx
72327 72077 72077     0  S       kqread   0xfffff80076229a00 nginx
72077     1 72077     0  Ss      pause    0xfffff8001f9270a8 nginx
86052     1 86052     0  Ss      (threaded)                  dpinger
100168                   S       uwait    0xfffff8001f908c00 dpinger
100209                   S       sbwait   0xfffff80076012144 dpinger
100210                   S       nanslp   0xffffffff829d1702 dpinger
100211                   S       nanslp   0xffffffff829d1702 dpinger
100212                   S       accept   0xfffff80076027a8c dpinger
85768     1 85768     0  Ss      (threaded)                  dpinger
100176                   S       uwait    0xfffff8001f829d00 dpinger
100205                   S       sbwait   0xfffff8001f9bf144 dpinger
100206                   S       nanslp   0xffffffff829d1702 dpinger
100207                   S       nanslp   0xffffffff829d1702 dpinger
100208                   S       accept   0xfffff8001fda23cc dpinger
85706     1 85706     0  Ss      (threaded)                  dpinger
100142                   S       uwait    0xfffff8001f281c00 dpinger
100201                   S       sbwait   0xfffff8001fda2b64 dpinger
100202                   S       nanslp   0xffffffff829d1701 dpinger
100203                   S       nanslp   0xffffffff829d1702 dpinger
100204                   S       accept   0xfffff8001fda0a8c dpinger
85250     1 85250     0  Ss      (threaded)                  dpinger
100180                   S       uwait    0xfffff8001f29e080 dpinger
100197                   S       sbwait   0xfffff8001fda1b64 dpinger
100198                   S       nanslp   0xffffffff829d1703 dpinger
100199                   S       nanslp   0xffffffff829d1700 dpinger
100200                   S       accept   0xfffff8001fda072c dpinger
84932     1 84932     0  Ss      (threaded)                  dpinger
100183                   S       uwait    0xfffff8006fff3900 dpinger
100193                   S       sbwait   0xfffff800760094a4 dpinger
100194                   S       nanslp   0xffffffff829d1700 dpinger
100195                   S       nanslp   0xffffffff829d1700 dpinger
100196                   S       accept   0xfffff8006ffe806c dpinger
61918     1 61918     0  Ss      bpf      0xfffff8001f8d3a00 filterlog
 8805   349   349     0  S       (threaded)                  php-fpm
100163                   S       piperd   0xfffff8019bf08be0 php-fpm
90201   349   349     0  S       lockf    0xfffff80121b5f080 php-fpm
 7329     1  7329     0  Ss      select   0xfffff8001f906ac0 sshd
  453     1   453     0  Ss      wait     0xfffff8001f80d000 devd
  390   388   388     0  S       kqread   0xfffff8001f91d000 check_reload_status
  388     1   388     0  Ss      kqread   0xfffff8001f933d00 check_reload_status
  351   349   349     0  S       nanslp   0xffffffff829d1701 php-fpm
  350   349   349     0  S       accept   0xfffff8001fb1aa8c php-fpm
  349     1   349     0  Ss      kqread   0xfffff8001f91c400 php-fpm
   63     0     0     0  DL      mdwait   0xfffff8001f270000 [md0]
   30     0     0     0  DL      syncer   0xffffffff82aca580 [syncer]
   29     0     0     0  DL      vlruwt   0xfffff8001f2f1000 [vnlru]
   28     0     0     0  DL      (threaded)                  [bufdaemon]
100128                   D       psleep   0xffffffff82ac9828 [bufdaemon]
100136                   D       sdflush  0xfffff8001f3e58e8 [/ worker]
   27     0     0     0  DL      -        0xffffffff82ac99fc [bufspacedaemon]
   26     0     0     0  DL      pgzero   0xffffffff82b0b568 [pagezero]
   25     0     0     0  DL      psleep   0xffffffff82b0b558 [vmdaemon]
   24     0     0     0  DL      (threaded)                  [pagedaemon]
100122                   D       psleep   0xffffffff82bb5d25 [dom0]
100124                   D       launds   0xffffffff82b079c4 [laundry: dom0]
100126                   D       umarcl   0xffffffff82b07338 [uma]
   23     0     0     0  DL      idle     0xfffff800083d3590 [enc_daemon0]
   22     0     0     0  DL      cooling  0xfffff8001e041558 [acpi_cooling0]
   21     0     0     0  DL      tzpoll   0xffffffff8276ae20 [acpi_thermal]
   20     0     0     0  DL      -        0xffffffff828a42d8 [rand_harvestq]
   19     0     0     0  DL      pftm     0xffffffff80f40140 [pf purge]
   18     0     0     0  DL      waiting_ 0xffffffff82bacbe0 [sctp_iterator]
   17     0     0     0  DL      -        0xffffffff82ac928c [soaiod4]
   16     0     0     0  DL      -        0xffffffff82ac928c [soaiod3]
    9     0     0     0  DL      -        0xffffffff82ac928c [soaiod2]
    8     0     0     0  DL      -        0xffffffff82ac928c [soaiod1]
   15     0     0     0  DL      (threaded)                  [usb]
100090                   D       -        0xfffffe0001272d10 [usbus0]
100091                   D       -        0xfffffe0001272d68 [usbus0]
100092                   D       -        0xfffffe0001272dc0 [usbus0]
100093                   D       -        0xfffffe0001272e18 [usbus0]
100094                   D       -        0xfffffe0001272e70 [usbus0]
100098                   D       -        0xfffffe0001290d10 [usbus1]
100099                   D       -        0xfffffe0001290d68 [usbus1]
100100                   D       -        0xfffffe0001290dc0 [usbus1]
100101                   D       -        0xfffffe0001290e18 [usbus1]
100102                   D       -        0xfffffe0001290e70 [usbus1]
    7     0     0     0  DL      (threaded)                  [cam]
100048                   D       -        0xffffffff82757000 [doneq0]
100120                   D       -        0xffffffff82756e48 [scanner]
    6     0     0     0  DL      crypto_r 0xfffff800084f7950 [crypto returns 3]
    5     0     0     0  DL      crypto_r 0xfffff800084f78f0 [crypto returns 2]
    4     0     0     0  DL      crypto_r 0xfffff800084f7890 [crypto returns 1]
    3     0     0     0  DL      crypto_r 0xfffff800084f7830 [crypto returns 0]
    2     0     0     0  DL      crypto_w 0xffffffff82b05e60 [crypto]
   14     0     0     0  DL      (threaded)                  [geom]
100035                   D       -        0xffffffff82b84c40 [g_event]
100036                   D       -        0xffffffff82b84c48 [g_up]
100037                   D       -        0xffffffff82b84c50 [g_down]
   13     0     0     0  DL      (threaded)                  [ng_queue]
100031                   D       sleep    0xffffffff8270f580 [ng_queue0]
100032                   D       sleep    0xffffffff8270f580 [ng_queue1]
100033                   D       sleep    0xffffffff8270f580 [ng_queue2]
100034                   D       sleep    0xffffffff8270f580 [ng_queue3]
   12     0     0     0  RL      (threaded)                  [intr]
100007                   I                                   [swi6: task queue]
100008                   I                                   [swi6: Giant taskq]
100011                   I                                   [swi5: fast taskq]
100022                   I                                   [swi1: netisr 0]
100023                   I                                   [swi1: netisr 1]
100024                   I                                   [swi1: netisr 2]
100025                   I                                   [swi1: netisr 3]
100026                   Run     CPU 0                       [swi4: clock (0)]
100027                   I                                   [swi4: clock (1)]
100028                   I                                   [swi4: clock (2)]
100029                   I                                   [swi4: clock (3)]
100030                   I                                   [swi3: vm]
100049                   I                                   [irq264: ixl0:aq]
100051                   I                                   [irq265: ixl0:q0]
100052                   I                                   [irq266: ixl0:q1]
100053                   I                                   [irq267: ixl0:q2]
100054                   I                                   [irq268: ixl0:q3]
100059                   I                                   [irq269: ixl1:aq]
100061                   I                                   [irq270: ixl1:q0]
100062                   I                                   [irq271: ixl1:q1]
100063                   I                                   [irq272: ixl1:q2]
100064                   I                                   [irq273: ixl1:q3]
100069                   I                                   [irq274: ixl2:aq]
100071                   I                                   [irq275: ixl2:q0]
100072                   I                                   [irq276: ixl2:q1]
100073                   I                                   [irq277: ixl2:q2]
100074                   I                                   [irq278: ixl2:q3]
100079                   I                                   [irq279: ixl3:aq]
100081                   I                                   [irq280: ixl3:q0]
100082                   I                                   [irq281: ixl3:q1]
100083                   I                                   [irq282: ixl3:q2]
100084                   I                                   [irq283: ixl3:q3]
100089                   I                                   [irq20: ehci0]
100095                   I                                   [irq284: bce0]
100096                   I                                   [irq285: bce1]
100097                   I                                   [irq23: ehci1]
100103                   I                                   [irq286: ahci0]
100111                   I                                   [swi1: pf send]
100112                   I                                   [swi1: pfsync]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
100005                   CanRun                              [idle: cpu2]
100006                   CanRun                              [idle: cpu3]
    1     0     1     0  SLs     wait     0xfffff800083d2590 [init]
   10     0     0     0  DL      audit_wo 0xffffffff82bb3dc0 [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin   0xffffffff82b84c88 [swapper]
100009                   D       -        0xfffff800083c1600 [thread taskq]
100010                   D       -        0xfffff800083c1400 [aiod_kick taskq]
100012                   D       -        0xfffff800083c0e00 [kqueue_ctx taskq]
100013                   D       -        0xfffff800083c0c00 [if_io_tqg_0]
100014                   D       -        0xfffff800083c0a00 [if_io_tqg_1]
100015                   D       -        0xfffff800083c0800 [if_io_tqg_2]
100016                   D       -        0xfffff800083c0600 [if_io_tqg_3]
100017                   D       -        0xfffff800083c0400 [softirq_0]
100018                   D       -        0xfffff800083c0200 [softirq_1]
100019                   D       -        0xfffff800083c0000 [softirq_2]
100020                   D       -        0xfffff80008413000 [softirq_3]
100021                   D       -        0xfffff8000841d200 [if_config_tqg_0]
100038                   D       -        0xfffff8000841ab00 [firmware taskq]
100039                   D       -        0xfffff8000841a100 [crypto_0]
100040                   D       -        0xfffff8000841a100 [crypto_1]
100041                   D       -        0xfffff8000841a100 [crypto_2]
100042                   D       -        0xfffff8000841a100 [crypto_3]
100050                   D       -        0xfffff800088fda00 [ixl0 aq]
100055                   D       -        0xfffff800088fd500 [ixl0 (que 0)]
100056                   D       -        0xfffff800088fd300 [ixl0 (que 1)]
100057                   D       -        0xfffff800088fd100 [ixl0 (que 2)]
100058                   D       -        0xfffff800088fce00 [ixl0 (que 3)]
100060                   D       -        0xfffff80008b6c700 [ixl1 aq]
100065                   D       -        0xfffff80008412d00 [ixl1 (que 0)]
100066                   D       -        0xfffff80008412b00 [ixl1 (que 1)]
100067                   D       -        0xfffff80008412900 [ixl1 (que 2)]
100068                   D       -        0xfffff80008412700 [ixl1 (que 3)]
100070                   D       -        0xfffff80008daf400 [ixl2 aq]
100075                   D       -        0xfffff80008b6c200 [ixl2 (que 0)]
100076                   D       -        0xfffff80008b6c000 [ixl2 (que 1)]
100077                   D       -        0xfffff80008b2cd00 [ixl2 (que 2)]
100078                   D       -        0xfffff80008b2cb00 [ixl2 (que 3)]
100080                   D       -        0xfffff80008fe0c00 [ixl3 aq]
100085                   D       -        0xfffff8001c084800 [ixl3 (que 0)]
100086                   D       -        0xfffff8001c084600 [ixl3 (que 1)]
100087                   D       -        0xfffff8001c084400 [ixl3 (que 2)]
100088                   D       -        0xfffff8001c084200 [ixl3 (que 3)]
100104                   D       -        0xfffff8001e062a00 [mca taskq]
100116                   D       -        0xfffff8001f23b600 [acpi_task_0]
100117                   D       -        0xfffff8001f23b600 [acpi_task_1]
100118                   D       -        0xfffff8001f23b600 [acpi_task_2]
100119                   D       -        0xfffff80008619100 [CAM taskq]
db:0:kdb.enter.default>  alltrace

Tracing command devd pid 86643 tid 100252 td 0xfffff80146e55620
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe045492edf0
ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe045492ee10
trap() at trap+0x3a/frame 0xfffffe045492ef20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe045492ef20
--- trap 0x13, rip = 0xffffffff811820e5, rsp = 0xfffffe0467776380, rbp = 0xfffffe0467776450 ---
pmap_remove_pages() at pmap_remove_pages+0x295/frame 0xfffffe0467776450
exec_new_vmspace() at exec_new_vmspace+0x1b5/frame 0xfffffe04677764c0
exec_elf64_imgact() at exec_elf64_imgact+0x931/frame 0xfffffe04677765b0
kern_execve() at kern_execve+0x77c/frame 0xfffffe0467776900
sys_execve() at sys_execve+0x4a/frame 0xfffffe0467776980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467776ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467776ab0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x449f2a, rsp = 0x7fffffffc8c8, rbp = 0x7fffffffc9c0 ---

Tracing command ifconfig pid 8390 tid 100290 td 0xfffff8019bb82000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467831660
mi_switch() at mi_switch+0xe6/frame 0xfffffe0467831690
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04678316f0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0467831730
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18c/frame 0xfffffe0467831790
seltdwait() at seltdwait+0x7b/frame 0xfffffe04678317d0
kern_poll() at kern_poll+0x3fd/frame 0xfffffe0467831960
sys_poll() at sys_poll+0x50/frame 0xfffffe0467831980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467831ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467831ab0
--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800d6f1ba, rsp = 0x7fffffffc7d8, rbp = 0x7fffffffcf00 ---

Tracing command sleep pid 98627 tid 100295 td 0xfffff8019bb06000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046784a760
mi_switch() at mi_switch+0xe6/frame 0xfffffe046784a790
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046784a7f0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe046784a830
_sleep() at _sleep+0x20c/frame 0xfffffe046784a8b0
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe046784a940
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe046784a980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046784aab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046784aab0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009055da, rsp = 0x7fffffffebc8, rbp = 0x7fffffffec10 ---

Tracing command sleep pid 13432 tid 100218 td 0xfffff80121dd1620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04676cc760
mi_switch() at mi_switch+0xe6/frame 0xfffffe04676cc790
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04676cc7f0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe04676cc830
_sleep() at _sleep+0x20c/frame 0xfffffe04676cc8b0
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe04676cc940
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe04676cc980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04676ccab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04676ccab0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009055da, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command dhcpd pid 138 tid 100289 td 0xfffff8019b0f5620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046782c5e0
mi_switch() at mi_switch+0xe6/frame 0xfffffe046782c610
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046782c670
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe046782c6b0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18c/frame 0xfffffe046782c710
seltdwait() at seltdwait+0x7b/frame 0xfffffe046782c750
kern_select() at kern_select+0x850/frame 0xfffffe046782c940
sys_select() at sys_select+0x56/frame 0xfffffe046782c980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046782cab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046782cab0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d63d2a, rsp = 0x7fffffffe7f8, rbp = 0x7fffffffe820 ---

Tracing command sh pid 52965 tid 100281 td 0xfffff8019ba1d000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467804670
mi_switch() at mi_switch+0xe6/frame 0xfffffe04678046a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467804700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0467804730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe0467804780
tty_wait() at tty_wait+0x1c/frame 0xfffffe04678047a0
ttydisc_read() at ttydisc_read+0x2ec/frame 0xfffffe0467804800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe0467804840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe04678048a0
dofileread() at dofileread+0xba/frame 0xfffffe04678048e0
kern_readv() at kern_readv+0x68/frame 0xfffffe0467804930
sys_read() at sys_read+0x86/frame 0xfffffe0467804980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467804ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467804ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bcdd4a, rsp = 0x7fffffffe768, rbp = 0x7fffffffe870 ---

Tracing command sh pid 52789 tid 100279 td 0xfffff8019ba1d620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677f8700
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677f8730
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677f8790
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677f87c0
_sleep() at _sleep+0x231/frame 0xfffffe04677f8840
pipe_read() at pipe_read+0x436/frame 0xfffffe04677f88a0
dofileread() at dofileread+0xba/frame 0xfffffe04677f88e0
kern_readv() at kern_readv+0x68/frame 0xfffffe04677f8930
sys_read() at sys_read+0x86/frame 0xfffffe04677f8980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677f8ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677f8ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bcdd4a, rsp = 0x7fffffffe7c8, rbp = 0x7fffffffe8d0 ---

Tracing command sh pid 52507 tid 100278 td 0xfffff8019ba50000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677f35c0
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677f35f0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677f3650
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677f3680
_sleep() at _sleep+0x231/frame 0xfffffe04677f3700
kern_wait6() at kern_wait6+0x4cd/frame 0xfffffe04677f3790
sys_wait4() at sys_wait4+0x78/frame 0xfffffe04677f3980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677f3ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677f3ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b6343a, rsp = 0x7fffffffe6c8, rbp = 0x7fffffffe780 ---

Tracing command sshg-blocker pid 52416 tid 100169 td 0xfffff8001f2b9000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04675d9700
mi_switch() at mi_switch+0xe6/frame 0xfffffe04675d9730
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04675d9790
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04675d97c0
_sleep() at _sleep+0x231/frame 0xfffffe04675d9840
pipe_read() at pipe_read+0x436/frame 0xfffffe04675d98a0
dofileread() at dofileread+0xba/frame 0xfffffe04675d98e0
kern_readv() at kern_readv+0x68/frame 0xfffffe04675d9930
sys_read() at sys_read+0x86/frame 0xfffffe04675d9980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04675d9ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04675d9ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b9ed4a, rsp = 0x7fffffffe648, rbp = 0x7fffffffe660 ---

Tracing command sshg-blocker pid 52416 tid 100280 td 0xfffff80121b7e620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677fd760
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677fd790
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677fd7f0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe04677fd830
_sleep() at _sleep+0x20c/frame 0xfffffe04677fd8b0
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe04677fd940
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe04677fd980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677fdab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677fdab0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b345da, rsp = 0x7fffdfffdef8, rbp = 0x7fffdfffdf20 ---

Tracing command sshg-parser pid 52283 tid 100275 td 0xfffff8019b12f620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677e4700
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677e4730
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677e4790
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677e47c0
_sleep() at _sleep+0x231/frame 0xfffffe04677e4840
pipe_read() at pipe_read+0x436/frame 0xfffffe04677e48a0
dofileread() at dofileread+0xba/frame 0xfffffe04677e48e0
kern_readv() at kern_readv+0x68/frame 0xfffffe04677e4930
sys_read() at sys_read+0x86/frame 0xfffffe04677e4980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677e4ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677e4ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x801100d4a, rsp = 0x7fffffffe6c8, rbp = 0x7fffffffe6e0 ---

Tracing command cat pid 52258 tid 100153 td 0xfffff8001f91a620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467589700
mi_switch() at mi_switch+0xe6/frame 0xfffffe0467589730
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467589790
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04675897c0
_sleep() at _sleep+0x231/frame 0xfffffe0467589840
pipe_read() at pipe_read+0x436/frame 0xfffffe04675898a0
dofileread() at dofileread+0xba/frame 0xfffffe04675898e0
kern_readv() at kern_readv+0x68/frame 0xfffffe0467589930
sys_read() at sys_read+0x86/frame 0xfffffe0467589980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467589ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467589ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800970d4a, rsp = 0x7fffffffe768, rbp = 0x7fffffffebf0 ---

Tracing command sh pid 51991 tid 100132 td 0xfffff8001f30d000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04675205c0
mi_switch() at mi_switch+0xe6/frame 0xfffffe04675205f0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467520650
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0467520680
_sleep() at _sleep+0x231/frame 0xfffffe0467520700
kern_wait6() at kern_wait6+0x4cd/frame 0xfffffe0467520790
sys_wait4() at sys_wait4+0x78/frame 0xfffffe0467520980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467520ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467520ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b6343a, rsp = 0x7fffffffe818, rbp = 0x7fffffffe8d0 ---

Tracing command sh pid 51844 tid 100165 td 0xfffff8001f982000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04675c55c0
mi_switch() at mi_switch+0xe6/frame 0xfffffe04675c55f0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04675c5650
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04675c5680
_sleep() at _sleep+0x231/frame 0xfffffe04675c5700
kern_wait6() at kern_wait6+0x4cd/frame 0xfffffe04675c5790
sys_wait4() at sys_wait4+0x78/frame 0xfffffe04675c5980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04675c5ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04675c5ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b6343a, rsp = 0x7fffffffe9e8, rbp = 0x7fffffffeaa0 ---

Tracing command getty pid 51562 tid 100191 td 0xfffff8007604e000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467645670
mi_switch() at mi_switch+0xe6/frame 0xfffffe04676456a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467645700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0467645730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe0467645780
tty_wait() at tty_wait+0x1c/frame 0xfffffe04676457a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe0467645800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe0467645840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe04676458a0
dofileread() at dofileread+0xba/frame 0xfffffe04676458e0
kern_readv() at kern_readv+0x68/frame 0xfffffe0467645930
sys_read() at sys_read+0x86/frame 0xfffffe0467645980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467645ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467645ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command getty pid 51297 tid 100243 td 0xfffff8019b031000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467749670
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677496a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467749700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0467749730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe0467749780
tty_wait() at tty_wait+0x1c/frame 0xfffffe04677497a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe0467749800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe0467749840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe04677498a0
dofileread() at dofileread+0xba/frame 0xfffffe04677498e0
kern_readv() at kern_readv+0x68/frame 0xfffffe0467749930
sys_read() at sys_read+0x86/frame 0xfffffe0467749980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467749ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467749ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command getty pid 51195 tid 100241 td 0xfffff80146e53000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046773f670
mi_switch() at mi_switch+0xe6/frame 0xfffffe046773f6a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046773f700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe046773f730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe046773f780
tty_wait() at tty_wait+0x1c/frame 0xfffffe046773f7a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe046773f800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe046773f840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe046773f8a0
dofileread() at dofileread+0xba/frame 0xfffffe046773f8e0
kern_readv() at kern_readv+0x68/frame 0xfffffe046773f930
sys_read() at sys_read+0x86/frame 0xfffffe046773f980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046773fab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046773fab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command getty pid 50972 tid 100138 td 0xfffff8001f84c000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046753e670
mi_switch() at mi_switch+0xe6/frame 0xfffffe046753e6a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046753e700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe046753e730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe046753e780
tty_wait() at tty_wait+0x1c/frame 0xfffffe046753e7a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe046753e800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe046753e840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe046753e8a0
dofileread() at dofileread+0xba/frame 0xfffffe046753e8e0
kern_readv() at kern_readv+0x68/frame 0xfffffe046753e930
sys_read() at sys_read+0x86/frame 0xfffffe046753e980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046753eab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046753eab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command getty pid 50832 tid 100251 td 0xfffff8019b030000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467771670
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677716a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467771700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0467771730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe0467771780
tty_wait() at tty_wait+0x1c/frame 0xfffffe04677717a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe0467771800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe0467771840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe04677718a0
dofileread() at dofileread+0xba/frame 0xfffffe04677718e0
kern_readv() at kern_readv+0x68/frame 0xfffffe0467771930
sys_read() at sys_read+0x86/frame 0xfffffe0467771980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467771ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467771ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command getty pid 50577 tid 100219 td 0xfffff80121dd1000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04676d1670
mi_switch() at mi_switch+0xe6/frame 0xfffffe04676d16a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04676d1700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04676d1730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe04676d1780
tty_wait() at tty_wait+0x1c/frame 0xfffffe04676d17a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe04676d1800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe04676d1840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe04676d18a0
dofileread() at dofileread+0xba/frame 0xfffffe04676d18e0
kern_readv() at kern_readv+0x68/frame 0xfffffe04676d1930
sys_read() at sys_read+0x86/frame 0xfffffe04676d1980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04676d1ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04676d1ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command getty pid 50360 tid 100223 td 0xfffff80121e48000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04676e5670
mi_switch() at mi_switch+0xe6/frame 0xfffffe04676e56a0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04676e5700
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04676e5730
_cv_wait_sig() at _cv_wait_sig+0x167/frame 0xfffffe04676e5780
tty_wait() at tty_wait+0x1c/frame 0xfffffe04676e57a0
ttydisc_read() at ttydisc_read+0x1f2/frame 0xfffffe04676e5800
ttydev_read() at ttydev_read+0x64/frame 0xfffffe04676e5840
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe04676e58a0
dofileread() at dofileread+0xba/frame 0xfffffe04676e58e0
kern_readv() at kern_readv+0x68/frame 0xfffffe04676e5930
sys_read() at sys_read+0x86/frame 0xfffffe04676e5980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04676e5ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04676e5ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b88d4a, rsp = 0x7fffffffec68, rbp = 0x7fffffffecb0 ---

Tracing command login pid 50253 tid 100249 td 0xfffff8019b0f4000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677675c0
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677675f0
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe0467767650
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0467767680
_sleep() at _sleep+0x231/frame 0xfffffe0467767700
kern_wait6() at kern_wait6+0x4cd/frame 0xfffffe0467767790
sys_wait4() at sys_wait4+0x78/frame 0xfffffe0467767980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467767ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467767ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d2a43a, rsp = 0x7fffffffebc8, rbp = 0x7fffffffed20 ---

Tracing command filterdns pid 98172 tid 100256 td 0xfffff80121bf3000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046778a720
mi_switch() at mi_switch+0xe6/frame 0xfffffe046778a750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046778a7b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe046778a7e0
_sleep() at _sleep+0x231/frame 0xfffffe046778a860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe046778a8c0
do_wait() at do_wait+0x427/frame 0xfffffe046778a940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe046778a980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046778aab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046778aab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdfffdeb8, rbp = 0x7fffdfffdee0 ---

Tracing command filterdns pid 98172 tid 100257 td 0xfffff80121bcf620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046778f720
mi_switch() at mi_switch+0xe6/frame 0xfffffe046778f750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046778f7b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe046778f7e0
_sleep() at _sleep+0x231/frame 0xfffffe046778f860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe046778f8c0
do_wait() at do_wait+0x427/frame 0xfffffe046778f940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe046778f980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046778fab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046778fab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdfdfceb8, rbp = 0x7fffdfdfcee0 ---

Tracing command filterdns pid 98172 tid 100258 td 0xfffff8019b057000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467794720
mi_switch() at mi_switch+0xe6/frame 0xfffffe0467794750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677947b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677947e0
_sleep() at _sleep+0x231/frame 0xfffffe0467794860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677948c0
do_wait() at do_wait+0x427/frame 0xfffffe0467794940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe0467794980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467794ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467794ab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdfbfbeb8, rbp = 0x7fffdfbfbee0 ---

Tracing command filterdns pid 98172 tid 100259 td 0xfffff8001fdf6620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe0467799720
mi_switch() at mi_switch+0xe6/frame 0xfffffe0467799750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677997b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677997e0
_sleep() at _sleep+0x231/frame 0xfffffe0467799860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677998c0
do_wait() at do_wait+0x427/frame 0xfffffe0467799940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe0467799980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0467799ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0467799ab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdf9faeb8, rbp = 0x7fffdf9faee0 ---

Tracing command filterdns pid 98172 tid 100260 td 0xfffff8001fdf6000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe046779e720
mi_switch() at mi_switch+0xe6/frame 0xfffffe046779e750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe046779e7b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe046779e7e0
_sleep() at _sleep+0x231/frame 0xfffffe046779e860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe046779e8c0
do_wait() at do_wait+0x427/frame 0xfffffe046779e940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe046779e980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe046779eab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe046779eab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdf7f9eb8, rbp = 0x7fffdf7f9ee0 ---

Tracing command filterdns pid 98172 tid 100261 td 0xfffff8019b061620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677a3720
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677a3750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677a37b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677a37e0
_sleep() at _sleep+0x231/frame 0xfffffe04677a3860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677a38c0
do_wait() at do_wait+0x427/frame 0xfffffe04677a3940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe04677a3980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677a3ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677a3ab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdf5f8eb8, rbp = 0x7fffdf5f8ee0 ---

Tracing command filterdns pid 98172 tid 100262 td 0xfffff8019b061000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677a8720
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677a8750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677a87b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677a87e0
_sleep() at _sleep+0x231/frame 0xfffffe04677a8860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677a88c0
do_wait() at do_wait+0x427/frame 0xfffffe04677a8940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe04677a8980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677a8ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677a8ab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdf3f7eb8, rbp = 0x7fffdf3f7ee0 ---

Tracing command filterdns pid 98172 tid 100263 td 0xfffff8019b7c6620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677ad720
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677ad750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677ad7b0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe04677ad7e0
_sleep() at _sleep+0x231/frame 0xfffffe04677ad860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677ad8c0
do_wait() at do_wait+0x427/frame 0xfffffe04677ad940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe04677ad980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677adab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677adab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdf1f6eb8, rbp = 0x7fffdf1f6ee0 ---

Tracing command filterdns pid 98172 tid 100264 td 0xfffff8019b7c6000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677b2710
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677b2740
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677b27a0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe04677b27e0
_sleep() at _sleep+0x20c/frame 0xfffffe04677b2860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677b28c0
do_wait() at do_wait+0x427/frame 0xfffffe04677b2940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe04677b2980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677b2ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677b2ab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdeff5e28, rbp = 0x7fffdeff5e50 ---

Tracing command filterdns pid 98172 tid 100265 td 0xfffff8019b12f000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677b7710
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677b7740
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677b77a0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe04677b77e0
_sleep() at _sleep+0x20c/frame 0xfffffe04677b7860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677b78c0
do_wait() at do_wait+0x427/frame 0xfffffe04677b7940
__umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x53/frame 0xfffffe04677b7980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe04677b7ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe04677b7ab0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x800840b2c, rsp = 0x7fffdedf4e28, rbp = 0x7fffdedf4e50 ---

Tracing command filterdns pid 98172 tid 100266 td 0xfffff8019b3d0620
sched_switch() at sched_switch+0x8ad/frame 0xfffffe04677bc710
mi_switch() at mi_switch+0xe6/frame 0xfffffe04677bc740
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe04677bc7a0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe04677bc7e0
_sleep() at _sleep+0x20c/frame 0xfffffe04677bc860
umtxq_sleep() at umtxq_sleep+0x143/frame 0xfffffe04677bc8c0
do_wait() at do_wait+0x427/frame 0xfffffe04677bc940
__umtx_op_wait_uint_private() at __umtx_op_wait_ui

Also available in: Atom PDF