Actions
Bug #1676
closed
CB
dead IPv6 gateway causes kernel panics
Bug #1676:
dead IPv6 gateway causes kernel panics
Added by Chris Buechler about 15 years ago. Updated over 14 years ago.
Start date:
07/13/2011
Due date:
% Done:
0%
Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.1-IPv6
Affected Architecture:
Description
It appears just having an IPv6 gateway configured that's unreachable will result in panics several times a day, even when that gateway is not in the routing table at all. Apparently just having apinger pinging an offline gateway will trigger a panic. Needs more investigation, but Seth has confirmed stability issues resolved after removing such an IPv6 gateway.
JP Updated by Jim Pingle about 15 years ago Actions #1
Definitely easy to reproduce with the right conditions, mine panics thusly:
- Home router with IPv6 connectivity via GIF tunnel(s). If a gif tunnel has a down gateway, there is no problem.
- Added two devices on my LAN as IPv6 gateways, for static routes.
- I carved some /64's out of my /48 from he.net and routed them to a VM and to my ALIX on the LAN.
- If I so much as reboot the VM that is a gateway and static route target, my main router panics within a minute or two of the VM being unreachable.
Bits from the textdump:
# cat version.txt
FreeBSD 8.1-RELEASE-p4 #1: Fri Jul 15 05:53:19 EDT 2011
sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
# tail -25 msgbuf.txt <118> Starting /usr/local/etc/rc.d/siproxd.sh... <118>done. <118>Bootup complete <6>pid 54184 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 32435 (filterdns), uid 0: exited on signal 11 (core dumped) <5>ovpnc1: link state changed to DOWN <5>ovpnc4: link state changed to DOWN <5>ovpnc4: link state changed to UP <5>ovpnc1: link state changed to UP <6>pid 2912 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 646 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 41638 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 14343 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 51567 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 38423 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 42119 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 14138 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 58146 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 31759 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 30753 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 7188 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 50572 (filterdns), uid 0: exited on signal 11 (core dumped) panic: sbappendaddr_locked cpuid = 0 KDB: enter: panic
Bits from ddb.txt
db:0:kdb.enter.panic> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.panic> show pcpu
cpuid = 0
dynamic pcpu = 0x53a680
curthread = 0xc546d500: pid 63976 "openvpn"
curpcb = 0xe59c5d90
fpcurthread = none
idlethread = 0xc498fa00: pid 11 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
db:0:kdb.enter.panic> bt
Tracing pid 63976 tid 64098 td 0xc546d500
kdb_enter(c0eca8d7,c0eca8d7,c0ecfdd7,e59c59b8,0,...) at kdb_enter+0x3a
panic(c0ecfdd7,c0a71fbb,c5635530,16,0,...) at panic+0x136
sbappendaddr_locked(c5310b98,e59c5a48,c53c8400,0,0,...) at sbappendaddr_locked+0x30
rip_append(e59c5a48,0,3b9aca00,1,0,...) at rip_append+0xfd
rip_input(c5215600,14,0,c0ee5450,1af,...) at rip_input+0x2cf
icmp_input(c5215600,14,12,0,0,...) at icmp_input+0x57f
ip_input(c5215600,44,c5215600,44,e59c5b80,...) at ip_input+0x7c3
netisr_dispatch_src(1,0,c5215600,e59c5bac,c0adcd4b,...) at netisr_dispatch_src+0x205
netisr_dispatch(1,c5215600,3,0,3,...) at netisr_dispatch+0x20
tunwrite(c54bba00,c54e3e80,4,c54cd000,e59c5bf4,...) at tunwrite+0x27b
giant_write(c54bba00,c54e3e80,4,0,0,...) at giant_write+0x89
devfs_write_f(c51460a8,c54e3e80,c498a400,0,c546d500,...) at devfs_write_f+0x7f
dofilewrite(c54e3e80,ffffffff,ffffffff,0,c51460a8,...) at dofilewrite+0x97
kern_writev(c546d500,d,c54e3e80,c54e3e80,0,...) at kern_writev+0x58
writev(c546d500,e59c5cf8,e59c5c98,c0a1a8d8,e59c5cd0,...) at writev+0x46
syscall(e59c5d38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (121, FreeBSD ELF32, writev), eip = 0x283a6c1b, esp = 0xbfbfe3bc, ebp = 0xbfbfe3f8 ---
db:0:kdb.enter.panic> ps
pid ppid pgrp uid state wmesg wchan cmd
4184 49934 56772 0 S nanslp 0xc1321424 sleep
42206 1 42206 0 Ss (threaded) filterdns
64075 S nanslp 0xc1321424 filterdns
64126 S uwait 0xc54e35c0 filterdns
49934 1 56772 0 S wait 0xc5606d48 sh
46812 1 46812 0 Rs apinger
34158 1 34158 0 Ss select 0xc54e3864 bsnmpd
23285 1 23285 0 Ss select 0xc546c1a4 rtadvd
21710 1 21710 1002 Ss select 0xc546b124 dhcpd
8700 1 8700 1002 Ss select 0xc50efda4 dhcpd
61147 1 60887 65534 S select 0xc571b4e4 dnsmasq
60276 1 60276 0 Ss kqread 0xc561de80 dhcpleases
32815 32451 32451 131 S kqread 0xc5135d80 ospfd
32630 32451 32451 131 S kqread 0xc54d9a80 ospfd
32451 1 32451 0 Ss kqread 0xc54d9e80 ospfd
36594 1 36594 0 Ss select 0xc54e34e4 ntpd
35441 1 254 123 S select 0xc54e3c64 ntpd
47424 1 47424 0 Ss select 0xc546cde4 openvpn
43053 1 43053 0 Ss select 0xc571b924 openvpn
31052 1 31052 0 Ss select 0xc571b5e4 racoon
17011 13744 17011 0 S+ ttyin 0xc4a42e70 sh
16795 13768 16795 0 S+ ttyin 0xc4a42870 sh
13768 12507 13768 0 S+ wait 0xc56c62a8 sh
13744 12170 13744 0 S+ wait 0xc5605d48 sh
13714 273 13714 0 Ss (threaded) sshlockout_pf
64145 S nanslp 0xc1321424 sshlockout_pf
64070 S piperd 0xc5150930 initial thread
12507 1 12507 0 Ss+ wait 0xc56c6aa0 login
12170 1 12170 0 Ss+ wait 0xc56c5aa0 login
1804 1 1804 0 Ss nanslp 0xc1321424 minicron
1370 1 1370 0 Ss nanslp 0xc1321424 minicron
961 1 961 0 Ss nanslp 0xc1321424 minicron
37905 1 37905 0 Ss nanslp 0xc1321424 cron
37449 1 37449 0 Ss select 0xc54e34a4 miniupnpd
37361 1 37361 0 Ss select 0xc54e30e4 powerd
58693 56772 56772 0 S accept 0xc55c99e6 php
58484 56772 56772 0 S accept 0xc55c99e6 php
57805 56196 56196 0 S accept 0xc55c984a php
57495 56196 56196 0 S accept 0xc55c984a php
56772 55892 56772 0 Ss wait 0xc5513000 initial thread
56196 55892 56196 0 Ss wait 0xc5513aa0 initial thread
55892 1 55836 0 S kqread 0xc5571680 lighttpd
31927 1 31927 0 Ss select 0xc54e31e4 inetd
1339 1 24 0 S+ piperd 0xc514fab8 logger
1266 1 24 0 S+ bpf 0xc54cb900 tcpdump
273 1 273 0 Ss select 0xc546c9e4 syslogd
63976 1 63976 0 Rs CPU 0 openvpn
59746 1 59746 0 Ss select 0xc50efd64 openvpn
55035 1 55035 0 Ss select 0xc546b164 openvpn
52931 1 52931 0 Ss select 0xc50efa64 openvpn
20209 1 20209 65 Ss select 0xc50ef9e4 dhclient
14772 1 14772 0 Ss select 0xc50ef924 dhclient
11779 1 11779 0 Ss select 0xc50ef7e4 dhcp6c
11499 1 11499 0 Ss select 0xc50ef724 sshd
7469 1 7469 0 Ss (threaded) mpd5
64078 S select 0xc4bac6a4 mpd5
267 1 267 0 Ss select 0xc50ee164 devd
256 254 254 0 S kqread 0xc5135900 check_reload_status
254 1 254 0 Ss kqread 0xc5135c80 check_reload_status
39 0 0 0 SL mdwait 0xc5127800 [md0]
23 0 0 0 SL flowclea 0xc13351a8 [flowcleaner]
22 0 0 0 SL sdflush 0xc134ff20 [softdepflush]
21 0 0 0 SL syncer 0xc1334f94 [syncer]
20 0 0 0 SL vlruwt 0xc50f32a8 [vnlru]
19 0 0 0 SL psleep 0xc1334cc8 [bufdaemon]
18 0 0 0 SL pollid 0xc132097c [idlepoll]
17 0 0 0 SL pgzero 0xc1350bf4 [pagezero]
16 0 0 0 SL psleep 0xc135081c [vmdaemon]
15 0 0 0 SL psleep 0xc13507e4 [pagedaemon]
9 0 0 0 SL ccb_scan 0xc12eb654 [xpt_thrd]
8 0 0 0 SL pftm 0xc04f9be0 [pfpurge]
7 0 0 0 SL waiting_ 0xc133c418 [sctp_iterator]
14 0 0 0 SL (threaded) usb
64051 D - 0xc4b78d0c [usbus4]
64050 D - 0xc4b78cdc [usbus4]
64049 D - 0xc4b78cac [usbus4]
64048 D - 0xc4b78c7c [usbus4]
64047 D - 0xc4b5ddac [usbus3]
64046 D - 0xc4b5dd7c [usbus3]
64045 D - 0xc4b5dd4c [usbus3]
64044 D - 0xc4b5dd1c [usbus3]
64043 D - 0xc4b4bdac [usbus2]
64042 D - 0xc4b4bd7c [usbus2]
64041 D - 0xc4b4bd4c [usbus2]
64040 D - 0xc4b4bd1c [usbus2]
64039 D - 0xc4b37dac [usbus1]
64038 D - 0xc4b37d7c [usbus1]
64037 D - 0xc4b37d4c [usbus1]
64036 D - 0xc4b37d1c [usbus1]
64035 D - 0xc4b24dac [usbus0]
64034 D - 0xc4b24d7c [usbus0]
64033 D - 0xc4b24d4c [usbus0]
64032 D - 0xc4b24d1c [usbus0]
6 0 0 0 SL crypto_r 0xc134f4cc [crypto returns]
5 0 0 0 SL crypto_w 0xc134f4a8 [crypto]
4 0 0 0 SL - 0xc131eb24 [g_down]
3 0 0 0 SL - 0xc131eb20 [g_up]
2 0 0 0 SL - 0xc131eb18 [g_event]
13 0 0 0 SL sleep 0xc12c1aa0 [ng_queue0]
12 0 0 0 LL (threaded) intr
64054 I [irq1: atkbd0]
64053 I [irq7: ppc0]
64052 I [swi0: uart uart]
64031 I [irq21: uhci0 uhci1*]
64030 I [irq15: ata1]
64029 I [irq14: ata0]
64028 I [irq20: atapci0]
64024 I [irq9: acpi0]
64022 I [swi5: +]
64020 I [swi2: cambio]
64016 I [swi6: task queue]
64015 I [swi6: Giant taskq]
64007 L *Giant 0xc4990080 [swi4: clock]
64006 I [swi3: vm]
64005 I [swi1: netisr 0]
11 0 0 0 RL [idle: cpu0]
1 0 1 0 SLs wait 0xc498dd48 [init]
10 0 0 0 SL audit_wo 0xc134f840 [audit]
0 0 0 0 SLs (threaded) kernel
64027 D - 0xc4b15680 [em2 taskq]
64026 D - 0xc4afbd80 [em1 taskq]
64025 D - 0xc4aef3c0 [em0 taskq]
64023 D - 0xc4a85e00 [thread taskq]
64021 D - 0xc4a86000 [kqueue taskq]
64019 D - 0xc4a862c0 [acpi_task_2]
64018 D - 0xc4a862c0 [acpi_task_1]
64017 D - 0xc4a862c0 [acpi_task_0]
64009 D - 0xc4974600 [firmware taskq]
64001 D sched 0xc131ec00 [swapper]
SM Updated by Seth Mos about 15 years ago Actions #2
the sbappendaddr_locked() is a function that I believe comes from our one shot dumps patch which is active for our 2.0 FreeBSD 8.1 builds.
Other panics I noted have been in sbdrop(). I prepped a new firewall at work and booted it without cables attached and it rebooted in 3 minutes flat without any traffic. Dell R310, 2GB ram, AMD64, 4 igb, 2 bce.
Ideally Ermal could have a look and see if he can find fault where this comes from.
Both Apinger and a lot of other things uses sockets which this function affects. I did a quick scan for 32 bit integer limits (ipv6 being 128 bit) but couldn't easily see a buffer overflow.
SM Updated by Seth Mos almost 15 years ago Actions #3
- Status changed from New to Feedback
Word is that the move to FreeBSD 9 will solve some of the issues as the ipsec socketbuffer patch will be gone.
SM Updated by Seth Mos over 14 years ago Actions #4
It appears to be resolved by upgrading base to 8.3. We'll need to wait a bit more to get a definitive statement but it appears working for Jim and me.
SM Updated by Seth Mos over 14 years ago Actions #5
- Status changed from Feedback to Resolved
Considering this resolved, seen no hangs in a month
Actions
Also available in: Atom