Actions
Bug #1676
closed
dead IPv6 gateway causes kernel panics
Start date:
07/13/2011
Due date:
% Done:
0%
Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.1-IPv6
Affected Architecture:
Description
It appears just having an IPv6 gateway configured that's unreachable will result in panics several times a day, even when that gateway is not in the routing table at all. Apparently just having apinger pinging an offline gateway will trigger a panic. Needs more investigation, but Seth has confirmed stability issues resolved after removing such an IPv6 gateway.
Updated by Jim Pingle over 13 years ago
Definitely easy to reproduce with the right conditions, mine panics thusly:
- Home router with IPv6 connectivity via GIF tunnel(s). If a gif tunnel has a down gateway, there is no problem.
- Added two devices on my LAN as IPv6 gateways, for static routes.
- I carved some /64's out of my /48 from he.net and routed them to a VM and to my ALIX on the LAN.
- If I so much as reboot the VM that is a gateway and static route target, my main router panics within a minute or two of the VM being unreachable.
Bits from the textdump:
# cat version.txt
FreeBSD 8.1-RELEASE-p4 #1: Fri Jul 15 05:53:19 EDT 2011
sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
# tail -25 msgbuf.txt <118> Starting /usr/local/etc/rc.d/siproxd.sh... <118>done. <118>Bootup complete <6>pid 54184 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 32435 (filterdns), uid 0: exited on signal 11 (core dumped) <5>ovpnc1: link state changed to DOWN <5>ovpnc4: link state changed to DOWN <5>ovpnc4: link state changed to UP <5>ovpnc1: link state changed to UP <6>pid 2912 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 646 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 41638 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 14343 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 51567 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 38423 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 42119 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 14138 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 58146 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 31759 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 30753 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 7188 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 50572 (filterdns), uid 0: exited on signal 11 (core dumped) panic: sbappendaddr_locked cpuid = 0 KDB: enter: panic
Bits from ddb.txt
db:0:kdb.enter.panic> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.panic> show pcpu
cpuid = 0
dynamic pcpu = 0x53a680
curthread = 0xc546d500: pid 63976 "openvpn"
curpcb = 0xe59c5d90
fpcurthread = none
idlethread = 0xc498fa00: pid 11 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
db:0:kdb.enter.panic> bt
Tracing pid 63976 tid 64098 td 0xc546d500
kdb_enter(c0eca8d7,c0eca8d7,c0ecfdd7,e59c59b8,0,...) at kdb_enter+0x3a
panic(c0ecfdd7,c0a71fbb,c5635530,16,0,...) at panic+0x136
sbappendaddr_locked(c5310b98,e59c5a48,c53c8400,0,0,...) at sbappendaddr_locked+0x30
rip_append(e59c5a48,0,3b9aca00,1,0,...) at rip_append+0xfd
rip_input(c5215600,14,0,c0ee5450,1af,...) at rip_input+0x2cf
icmp_input(c5215600,14,12,0,0,...) at icmp_input+0x57f
ip_input(c5215600,44,c5215600,44,e59c5b80,...) at ip_input+0x7c3
netisr_dispatch_src(1,0,c5215600,e59c5bac,c0adcd4b,...) at netisr_dispatch_src+0x205
netisr_dispatch(1,c5215600,3,0,3,...) at netisr_dispatch+0x20
tunwrite(c54bba00,c54e3e80,4,c54cd000,e59c5bf4,...) at tunwrite+0x27b
giant_write(c54bba00,c54e3e80,4,0,0,...) at giant_write+0x89
devfs_write_f(c51460a8,c54e3e80,c498a400,0,c546d500,...) at devfs_write_f+0x7f
dofilewrite(c54e3e80,ffffffff,ffffffff,0,c51460a8,...) at dofilewrite+0x97
kern_writev(c546d500,d,c54e3e80,c54e3e80,0,...) at kern_writev+0x58
writev(c546d500,e59c5cf8,e59c5c98,c0a1a8d8,e59c5cd0,...) at writev+0x46
syscall(e59c5d38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (121, FreeBSD ELF32, writev), eip = 0x283a6c1b, esp = 0xbfbfe3bc, ebp = 0xbfbfe3f8 ---
db:0:kdb.enter.panic> ps
pid ppid pgrp uid state wmesg wchan cmd
4184 49934 56772 0 S nanslp 0xc1321424 sleep
42206 1 42206 0 Ss (threaded) filterdns
64075 S nanslp 0xc1321424 filterdns
64126 S uwait 0xc54e35c0 filterdns
49934 1 56772 0 S wait 0xc5606d48 sh
46812 1 46812 0 Rs apinger
34158 1 34158 0 Ss select 0xc54e3864 bsnmpd
23285 1 23285 0 Ss select 0xc546c1a4 rtadvd
21710 1 21710 1002 Ss select 0xc546b124 dhcpd
8700 1 8700 1002 Ss select 0xc50efda4 dhcpd
61147 1 60887 65534 S select 0xc571b4e4 dnsmasq
60276 1 60276 0 Ss kqread 0xc561de80 dhcpleases
32815 32451 32451 131 S kqread 0xc5135d80 ospfd
32630 32451 32451 131 S kqread 0xc54d9a80 ospfd
32451 1 32451 0 Ss kqread 0xc54d9e80 ospfd
36594 1 36594 0 Ss select 0xc54e34e4 ntpd
35441 1 254 123 S select 0xc54e3c64 ntpd
47424 1 47424 0 Ss select 0xc546cde4 openvpn
43053 1 43053 0 Ss select 0xc571b924 openvpn
31052 1 31052 0 Ss select 0xc571b5e4 racoon
17011 13744 17011 0 S+ ttyin 0xc4a42e70 sh
16795 13768 16795 0 S+ ttyin 0xc4a42870 sh
13768 12507 13768 0 S+ wait 0xc56c62a8 sh
13744 12170 13744 0 S+ wait 0xc5605d48 sh
13714 273 13714 0 Ss (threaded) sshlockout_pf
64145 S nanslp 0xc1321424 sshlockout_pf
64070 S piperd 0xc5150930 initial thread
12507 1 12507 0 Ss+ wait 0xc56c6aa0 login
12170 1 12170 0 Ss+ wait 0xc56c5aa0 login
1804 1 1804 0 Ss nanslp 0xc1321424 minicron
1370 1 1370 0 Ss nanslp 0xc1321424 minicron
961 1 961 0 Ss nanslp 0xc1321424 minicron
37905 1 37905 0 Ss nanslp 0xc1321424 cron
37449 1 37449 0 Ss select 0xc54e34a4 miniupnpd
37361 1 37361 0 Ss select 0xc54e30e4 powerd
58693 56772 56772 0 S accept 0xc55c99e6 php
58484 56772 56772 0 S accept 0xc55c99e6 php
57805 56196 56196 0 S accept 0xc55c984a php
57495 56196 56196 0 S accept 0xc55c984a php
56772 55892 56772 0 Ss wait 0xc5513000 initial thread
56196 55892 56196 0 Ss wait 0xc5513aa0 initial thread
55892 1 55836 0 S kqread 0xc5571680 lighttpd
31927 1 31927 0 Ss select 0xc54e31e4 inetd
1339 1 24 0 S+ piperd 0xc514fab8 logger
1266 1 24 0 S+ bpf 0xc54cb900 tcpdump
273 1 273 0 Ss select 0xc546c9e4 syslogd
63976 1 63976 0 Rs CPU 0 openvpn
59746 1 59746 0 Ss select 0xc50efd64 openvpn
55035 1 55035 0 Ss select 0xc546b164 openvpn
52931 1 52931 0 Ss select 0xc50efa64 openvpn
20209 1 20209 65 Ss select 0xc50ef9e4 dhclient
14772 1 14772 0 Ss select 0xc50ef924 dhclient
11779 1 11779 0 Ss select 0xc50ef7e4 dhcp6c
11499 1 11499 0 Ss select 0xc50ef724 sshd
7469 1 7469 0 Ss (threaded) mpd5
64078 S select 0xc4bac6a4 mpd5
267 1 267 0 Ss select 0xc50ee164 devd
256 254 254 0 S kqread 0xc5135900 check_reload_status
254 1 254 0 Ss kqread 0xc5135c80 check_reload_status
39 0 0 0 SL mdwait 0xc5127800 [md0]
23 0 0 0 SL flowclea 0xc13351a8 [flowcleaner]
22 0 0 0 SL sdflush 0xc134ff20 [softdepflush]
21 0 0 0 SL syncer 0xc1334f94 [syncer]
20 0 0 0 SL vlruwt 0xc50f32a8 [vnlru]
19 0 0 0 SL psleep 0xc1334cc8 [bufdaemon]
18 0 0 0 SL pollid 0xc132097c [idlepoll]
17 0 0 0 SL pgzero 0xc1350bf4 [pagezero]
16 0 0 0 SL psleep 0xc135081c [vmdaemon]
15 0 0 0 SL psleep 0xc13507e4 [pagedaemon]
9 0 0 0 SL ccb_scan 0xc12eb654 [xpt_thrd]
8 0 0 0 SL pftm 0xc04f9be0 [pfpurge]
7 0 0 0 SL waiting_ 0xc133c418 [sctp_iterator]
14 0 0 0 SL (threaded) usb
64051 D - 0xc4b78d0c [usbus4]
64050 D - 0xc4b78cdc [usbus4]
64049 D - 0xc4b78cac [usbus4]
64048 D - 0xc4b78c7c [usbus4]
64047 D - 0xc4b5ddac [usbus3]
64046 D - 0xc4b5dd7c [usbus3]
64045 D - 0xc4b5dd4c [usbus3]
64044 D - 0xc4b5dd1c [usbus3]
64043 D - 0xc4b4bdac [usbus2]
64042 D - 0xc4b4bd7c [usbus2]
64041 D - 0xc4b4bd4c [usbus2]
64040 D - 0xc4b4bd1c [usbus2]
64039 D - 0xc4b37dac [usbus1]
64038 D - 0xc4b37d7c [usbus1]
64037 D - 0xc4b37d4c [usbus1]
64036 D - 0xc4b37d1c [usbus1]
64035 D - 0xc4b24dac [usbus0]
64034 D - 0xc4b24d7c [usbus0]
64033 D - 0xc4b24d4c [usbus0]
64032 D - 0xc4b24d1c [usbus0]
6 0 0 0 SL crypto_r 0xc134f4cc [crypto returns]
5 0 0 0 SL crypto_w 0xc134f4a8 [crypto]
4 0 0 0 SL - 0xc131eb24 [g_down]
3 0 0 0 SL - 0xc131eb20 [g_up]
2 0 0 0 SL - 0xc131eb18 [g_event]
13 0 0 0 SL sleep 0xc12c1aa0 [ng_queue0]
12 0 0 0 LL (threaded) intr
64054 I [irq1: atkbd0]
64053 I [irq7: ppc0]
64052 I [swi0: uart uart]
64031 I [irq21: uhci0 uhci1*]
64030 I [irq15: ata1]
64029 I [irq14: ata0]
64028 I [irq20: atapci0]
64024 I [irq9: acpi0]
64022 I [swi5: +]
64020 I [swi2: cambio]
64016 I [swi6: task queue]
64015 I [swi6: Giant taskq]
64007 L *Giant 0xc4990080 [swi4: clock]
64006 I [swi3: vm]
64005 I [swi1: netisr 0]
11 0 0 0 RL [idle: cpu0]
1 0 1 0 SLs wait 0xc498dd48 [init]
10 0 0 0 SL audit_wo 0xc134f840 [audit]
0 0 0 0 SLs (threaded) kernel
64027 D - 0xc4b15680 [em2 taskq]
64026 D - 0xc4afbd80 [em1 taskq]
64025 D - 0xc4aef3c0 [em0 taskq]
64023 D - 0xc4a85e00 [thread taskq]
64021 D - 0xc4a86000 [kqueue taskq]
64019 D - 0xc4a862c0 [acpi_task_2]
64018 D - 0xc4a862c0 [acpi_task_1]
64017 D - 0xc4a862c0 [acpi_task_0]
64009 D - 0xc4974600 [firmware taskq]
64001 D sched 0xc131ec00 [swapper]
Updated by Seth Mos over 13 years ago
the sbappendaddr_locked() is a function that I believe comes from our one shot dumps patch which is active for our 2.0 FreeBSD 8.1 builds.
Other panics I noted have been in sbdrop(). I prepped a new firewall at work and booted it without cables attached and it rebooted in 3 minutes flat without any traffic. Dell R310, 2GB ram, AMD64, 4 igb, 2 bce.
Ideally Ermal could have a look and see if he can find fault where this comes from.
Both Apinger and a lot of other things uses sockets which this function affects. I did a quick scan for 32 bit integer limits (ipv6 being 128 bit) but couldn't easily see a buffer overflow.
Updated by Seth Mos about 13 years ago
- Status changed from New to Feedback
Word is that the move to FreeBSD 9 will solve some of the issues as the ipsec socketbuffer patch will be gone.
Updated by Seth Mos almost 13 years ago
It appears to be resolved by upgrading base to 8.3. We'll need to wait a bit more to get a definitive statement but it appears working for Jim and me.
Updated by Seth Mos over 12 years ago
- Status changed from Feedback to Resolved
Considering this resolved, seen no hangs in a month
Actions