Bug #1676
closeddead IPv6 gateway causes kernel panics
Added by Chris Buechler about 13 years ago. Updated over 12 years ago.
0%
Description
It appears just having an IPv6 gateway configured that's unreachable will result in panics several times a day, even when that gateway is not in the routing table at all. Apparently just having apinger pinging an offline gateway will trigger a panic. Needs more investigation, but Seth has confirmed stability issues resolved after removing such an IPv6 gateway.
Updated by Jim Pingle about 13 years ago
Definitely easy to reproduce with the right conditions, mine panics thusly:
- Home router with IPv6 connectivity via GIF tunnel(s). If a gif tunnel has a down gateway, there is no problem.
- Added two devices on my LAN as IPv6 gateways, for static routes.
- I carved some /64's out of my /48 from he.net and routed them to a VM and to my ALIX on the LAN.
- If I so much as reboot the VM that is a gateway and static route target, my main router panics within a minute or two of the VM being unreachable.
Bits from the textdump:
# cat version.txt FreeBSD 8.1-RELEASE-p4 #1: Fri Jul 15 05:53:19 EDT 2011 sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
# tail -25 msgbuf.txt <118> Starting /usr/local/etc/rc.d/siproxd.sh... <118>done. <118>Bootup complete <6>pid 54184 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 32435 (filterdns), uid 0: exited on signal 11 (core dumped) <5>ovpnc1: link state changed to DOWN <5>ovpnc4: link state changed to DOWN <5>ovpnc4: link state changed to UP <5>ovpnc1: link state changed to UP <6>pid 2912 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 646 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 41638 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 14343 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 51567 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 38423 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 42119 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 14138 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 58146 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 31759 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 30753 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 7188 (filterdns), uid 0: exited on signal 11 (core dumped) <6>pid 50572 (filterdns), uid 0: exited on signal 11 (core dumped) panic: sbappendaddr_locked cpuid = 0 KDB: enter: panic
Bits from ddb.txt
db:0:kdb.enter.panic> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.panic> show pcpu cpuid = 0 dynamic pcpu = 0x53a680 curthread = 0xc546d500: pid 63976 "openvpn" curpcb = 0xe59c5d90 fpcurthread = none idlethread = 0xc498fa00: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 db:0:kdb.enter.panic> bt Tracing pid 63976 tid 64098 td 0xc546d500 kdb_enter(c0eca8d7,c0eca8d7,c0ecfdd7,e59c59b8,0,...) at kdb_enter+0x3a panic(c0ecfdd7,c0a71fbb,c5635530,16,0,...) at panic+0x136 sbappendaddr_locked(c5310b98,e59c5a48,c53c8400,0,0,...) at sbappendaddr_locked+0x30 rip_append(e59c5a48,0,3b9aca00,1,0,...) at rip_append+0xfd rip_input(c5215600,14,0,c0ee5450,1af,...) at rip_input+0x2cf icmp_input(c5215600,14,12,0,0,...) at icmp_input+0x57f ip_input(c5215600,44,c5215600,44,e59c5b80,...) at ip_input+0x7c3 netisr_dispatch_src(1,0,c5215600,e59c5bac,c0adcd4b,...) at netisr_dispatch_src+0x205 netisr_dispatch(1,c5215600,3,0,3,...) at netisr_dispatch+0x20 tunwrite(c54bba00,c54e3e80,4,c54cd000,e59c5bf4,...) at tunwrite+0x27b giant_write(c54bba00,c54e3e80,4,0,0,...) at giant_write+0x89 devfs_write_f(c51460a8,c54e3e80,c498a400,0,c546d500,...) at devfs_write_f+0x7f dofilewrite(c54e3e80,ffffffff,ffffffff,0,c51460a8,...) at dofilewrite+0x97 kern_writev(c546d500,d,c54e3e80,c54e3e80,0,...) at kern_writev+0x58 writev(c546d500,e59c5cf8,e59c5c98,c0a1a8d8,e59c5cd0,...) at writev+0x46 syscall(e59c5d38) at syscall+0x2d3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (121, FreeBSD ELF32, writev), eip = 0x283a6c1b, esp = 0xbfbfe3bc, ebp = 0xbfbfe3f8 --- db:0:kdb.enter.panic> ps pid ppid pgrp uid state wmesg wchan cmd 4184 49934 56772 0 S nanslp 0xc1321424 sleep 42206 1 42206 0 Ss (threaded) filterdns 64075 S nanslp 0xc1321424 filterdns 64126 S uwait 0xc54e35c0 filterdns 49934 1 56772 0 S wait 0xc5606d48 sh 46812 1 46812 0 Rs apinger 34158 1 34158 0 Ss select 0xc54e3864 bsnmpd 23285 1 23285 0 Ss select 0xc546c1a4 rtadvd 21710 1 21710 1002 Ss select 0xc546b124 dhcpd 8700 1 8700 1002 Ss select 0xc50efda4 dhcpd 61147 1 60887 65534 S select 0xc571b4e4 dnsmasq 60276 1 60276 0 Ss kqread 0xc561de80 dhcpleases 32815 32451 32451 131 S kqread 0xc5135d80 ospfd 32630 32451 32451 131 S kqread 0xc54d9a80 ospfd 32451 1 32451 0 Ss kqread 0xc54d9e80 ospfd 36594 1 36594 0 Ss select 0xc54e34e4 ntpd 35441 1 254 123 S select 0xc54e3c64 ntpd 47424 1 47424 0 Ss select 0xc546cde4 openvpn 43053 1 43053 0 Ss select 0xc571b924 openvpn 31052 1 31052 0 Ss select 0xc571b5e4 racoon 17011 13744 17011 0 S+ ttyin 0xc4a42e70 sh 16795 13768 16795 0 S+ ttyin 0xc4a42870 sh 13768 12507 13768 0 S+ wait 0xc56c62a8 sh 13744 12170 13744 0 S+ wait 0xc5605d48 sh 13714 273 13714 0 Ss (threaded) sshlockout_pf 64145 S nanslp 0xc1321424 sshlockout_pf 64070 S piperd 0xc5150930 initial thread 12507 1 12507 0 Ss+ wait 0xc56c6aa0 login 12170 1 12170 0 Ss+ wait 0xc56c5aa0 login 1804 1 1804 0 Ss nanslp 0xc1321424 minicron 1370 1 1370 0 Ss nanslp 0xc1321424 minicron 961 1 961 0 Ss nanslp 0xc1321424 minicron 37905 1 37905 0 Ss nanslp 0xc1321424 cron 37449 1 37449 0 Ss select 0xc54e34a4 miniupnpd 37361 1 37361 0 Ss select 0xc54e30e4 powerd 58693 56772 56772 0 S accept 0xc55c99e6 php 58484 56772 56772 0 S accept 0xc55c99e6 php 57805 56196 56196 0 S accept 0xc55c984a php 57495 56196 56196 0 S accept 0xc55c984a php 56772 55892 56772 0 Ss wait 0xc5513000 initial thread 56196 55892 56196 0 Ss wait 0xc5513aa0 initial thread 55892 1 55836 0 S kqread 0xc5571680 lighttpd 31927 1 31927 0 Ss select 0xc54e31e4 inetd 1339 1 24 0 S+ piperd 0xc514fab8 logger 1266 1 24 0 S+ bpf 0xc54cb900 tcpdump 273 1 273 0 Ss select 0xc546c9e4 syslogd 63976 1 63976 0 Rs CPU 0 openvpn 59746 1 59746 0 Ss select 0xc50efd64 openvpn 55035 1 55035 0 Ss select 0xc546b164 openvpn 52931 1 52931 0 Ss select 0xc50efa64 openvpn 20209 1 20209 65 Ss select 0xc50ef9e4 dhclient 14772 1 14772 0 Ss select 0xc50ef924 dhclient 11779 1 11779 0 Ss select 0xc50ef7e4 dhcp6c 11499 1 11499 0 Ss select 0xc50ef724 sshd 7469 1 7469 0 Ss (threaded) mpd5 64078 S select 0xc4bac6a4 mpd5 267 1 267 0 Ss select 0xc50ee164 devd 256 254 254 0 S kqread 0xc5135900 check_reload_status 254 1 254 0 Ss kqread 0xc5135c80 check_reload_status 39 0 0 0 SL mdwait 0xc5127800 [md0] 23 0 0 0 SL flowclea 0xc13351a8 [flowcleaner] 22 0 0 0 SL sdflush 0xc134ff20 [softdepflush] 21 0 0 0 SL syncer 0xc1334f94 [syncer] 20 0 0 0 SL vlruwt 0xc50f32a8 [vnlru] 19 0 0 0 SL psleep 0xc1334cc8 [bufdaemon] 18 0 0 0 SL pollid 0xc132097c [idlepoll] 17 0 0 0 SL pgzero 0xc1350bf4 [pagezero] 16 0 0 0 SL psleep 0xc135081c [vmdaemon] 15 0 0 0 SL psleep 0xc13507e4 [pagedaemon] 9 0 0 0 SL ccb_scan 0xc12eb654 [xpt_thrd] 8 0 0 0 SL pftm 0xc04f9be0 [pfpurge] 7 0 0 0 SL waiting_ 0xc133c418 [sctp_iterator] 14 0 0 0 SL (threaded) usb 64051 D - 0xc4b78d0c [usbus4] 64050 D - 0xc4b78cdc [usbus4] 64049 D - 0xc4b78cac [usbus4] 64048 D - 0xc4b78c7c [usbus4] 64047 D - 0xc4b5ddac [usbus3] 64046 D - 0xc4b5dd7c [usbus3] 64045 D - 0xc4b5dd4c [usbus3] 64044 D - 0xc4b5dd1c [usbus3] 64043 D - 0xc4b4bdac [usbus2] 64042 D - 0xc4b4bd7c [usbus2] 64041 D - 0xc4b4bd4c [usbus2] 64040 D - 0xc4b4bd1c [usbus2] 64039 D - 0xc4b37dac [usbus1] 64038 D - 0xc4b37d7c [usbus1] 64037 D - 0xc4b37d4c [usbus1] 64036 D - 0xc4b37d1c [usbus1] 64035 D - 0xc4b24dac [usbus0] 64034 D - 0xc4b24d7c [usbus0] 64033 D - 0xc4b24d4c [usbus0] 64032 D - 0xc4b24d1c [usbus0] 6 0 0 0 SL crypto_r 0xc134f4cc [crypto returns] 5 0 0 0 SL crypto_w 0xc134f4a8 [crypto] 4 0 0 0 SL - 0xc131eb24 [g_down] 3 0 0 0 SL - 0xc131eb20 [g_up] 2 0 0 0 SL - 0xc131eb18 [g_event] 13 0 0 0 SL sleep 0xc12c1aa0 [ng_queue0] 12 0 0 0 LL (threaded) intr 64054 I [irq1: atkbd0] 64053 I [irq7: ppc0] 64052 I [swi0: uart uart] 64031 I [irq21: uhci0 uhci1*] 64030 I [irq15: ata1] 64029 I [irq14: ata0] 64028 I [irq20: atapci0] 64024 I [irq9: acpi0] 64022 I [swi5: +] 64020 I [swi2: cambio] 64016 I [swi6: task queue] 64015 I [swi6: Giant taskq] 64007 L *Giant 0xc4990080 [swi4: clock] 64006 I [swi3: vm] 64005 I [swi1: netisr 0] 11 0 0 0 RL [idle: cpu0] 1 0 1 0 SLs wait 0xc498dd48 [init] 10 0 0 0 SL audit_wo 0xc134f840 [audit] 0 0 0 0 SLs (threaded) kernel 64027 D - 0xc4b15680 [em2 taskq] 64026 D - 0xc4afbd80 [em1 taskq] 64025 D - 0xc4aef3c0 [em0 taskq] 64023 D - 0xc4a85e00 [thread taskq] 64021 D - 0xc4a86000 [kqueue taskq] 64019 D - 0xc4a862c0 [acpi_task_2] 64018 D - 0xc4a862c0 [acpi_task_1] 64017 D - 0xc4a862c0 [acpi_task_0] 64009 D - 0xc4974600 [firmware taskq] 64001 D sched 0xc131ec00 [swapper]
Updated by Seth Mos almost 13 years ago
the sbappendaddr_locked() is a function that I believe comes from our one shot dumps patch which is active for our 2.0 FreeBSD 8.1 builds.
Other panics I noted have been in sbdrop(). I prepped a new firewall at work and booted it without cables attached and it rebooted in 3 minutes flat without any traffic. Dell R310, 2GB ram, AMD64, 4 igb, 2 bce.
Ideally Ermal could have a look and see if he can find fault where this comes from.
Both Apinger and a lot of other things uses sockets which this function affects. I did a quick scan for 32 bit integer limits (ipv6 being 128 bit) but couldn't easily see a buffer overflow.
Updated by Seth Mos almost 13 years ago
- Status changed from New to Feedback
Word is that the move to FreeBSD 9 will solve some of the issues as the ipsec socketbuffer patch will be gone.
Updated by Seth Mos over 12 years ago
It appears to be resolved by upgrading base to 8.3. We'll need to wait a bit more to get a definitive statement but it appears working for Jim and me.
Updated by Seth Mos over 12 years ago
- Status changed from Feedback to Resolved
Considering this resolved, seen no hangs in a month