Project

General

Profile

Actions

Bug #1676

closed

dead IPv6 gateway causes kernel panics

Added by Chris Buechler over 10 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Operating System
Target version:
Start date:
07/13/2011
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:
Affected Version:
2.1-IPv6
Affected Architecture:

Description

It appears just having an IPv6 gateway configured that's unreachable will result in panics several times a day, even when that gateway is not in the routing table at all. Apparently just having apinger pinging an offline gateway will trigger a panic. Needs more investigation, but Seth has confirmed stability issues resolved after removing such an IPv6 gateway.

Actions #1

Updated by Jim Pingle over 10 years ago

Definitely easy to reproduce with the right conditions, mine panics thusly:

  • Home router with IPv6 connectivity via GIF tunnel(s). If a gif tunnel has a down gateway, there is no problem.
  • Added two devices on my LAN as IPv6 gateways, for static routes.
  • I carved some /64's out of my /48 from he.net and routed them to a VM and to my ALIX on the LAN.
  • If I so much as reboot the VM that is a gateway and static route target, my main router panics within a minute or two of the VM being unreachable.

Bits from the textdump:

# cat version.txt 
FreeBSD 8.1-RELEASE-p4 #1: Fri Jul 15 05:53:19 EDT 2011
    sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
# tail -25 msgbuf.txt 
<118> Starting /usr/local/etc/rc.d/siproxd.sh...
<118>done.
<118>Bootup complete
<6>pid 54184 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 32435 (filterdns), uid 0: exited on signal 11 (core dumped)
<5>ovpnc1: link state changed to DOWN
<5>ovpnc4: link state changed to DOWN
<5>ovpnc4: link state changed to UP
<5>ovpnc1: link state changed to UP
<6>pid 2912 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 646 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 41638 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 14343 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 51567 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 38423 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 42119 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 14138 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 58146 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 31759 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 30753 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 7188 (filterdns), uid 0: exited on signal 11 (core dumped)
<6>pid 50572 (filterdns), uid 0: exited on signal 11 (core dumped)
panic: sbappendaddr_locked
cpuid = 0
KDB: enter: panic

Bits from ddb.txt

db:0:kdb.enter.panic>  run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks>  show alllocks
No such command
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.panic>  show pcpu
cpuid        = 0
dynamic pcpu    = 0x53a680
curthread    = 0xc546d500: pid 63976 "openvpn" 
curpcb       = 0xe59c5d90
fpcurthread  = none
idlethread   = 0xc498fa00: pid 11 "idle: cpu0" 
APIC ID      = 0
currentldt   = 0x50
db:0:kdb.enter.panic>  bt
Tracing pid 63976 tid 64098 td 0xc546d500
kdb_enter(c0eca8d7,c0eca8d7,c0ecfdd7,e59c59b8,0,...) at kdb_enter+0x3a
panic(c0ecfdd7,c0a71fbb,c5635530,16,0,...) at panic+0x136
sbappendaddr_locked(c5310b98,e59c5a48,c53c8400,0,0,...) at sbappendaddr_locked+0x30
rip_append(e59c5a48,0,3b9aca00,1,0,...) at rip_append+0xfd
rip_input(c5215600,14,0,c0ee5450,1af,...) at rip_input+0x2cf
icmp_input(c5215600,14,12,0,0,...) at icmp_input+0x57f
ip_input(c5215600,44,c5215600,44,e59c5b80,...) at ip_input+0x7c3
netisr_dispatch_src(1,0,c5215600,e59c5bac,c0adcd4b,...) at netisr_dispatch_src+0x205
netisr_dispatch(1,c5215600,3,0,3,...) at netisr_dispatch+0x20
tunwrite(c54bba00,c54e3e80,4,c54cd000,e59c5bf4,...) at tunwrite+0x27b
giant_write(c54bba00,c54e3e80,4,0,0,...) at giant_write+0x89
devfs_write_f(c51460a8,c54e3e80,c498a400,0,c546d500,...) at devfs_write_f+0x7f
dofilewrite(c54e3e80,ffffffff,ffffffff,0,c51460a8,...) at dofilewrite+0x97
kern_writev(c546d500,d,c54e3e80,c54e3e80,0,...) at kern_writev+0x58
writev(c546d500,e59c5cf8,e59c5c98,c0a1a8d8,e59c5cd0,...) at writev+0x46
syscall(e59c5d38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (121, FreeBSD ELF32, writev), eip = 0x283a6c1b, esp = 0xbfbfe3bc, ebp = 0xbfbfe3f8 ---
db:0:kdb.enter.panic>  ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
 4184 49934 56772     0  S       nanslp   0xc1321424 sleep
42206     1 42206     0  Ss      (threaded)          filterdns
 64075                   S       nanslp   0xc1321424 filterdns
 64126                   S       uwait    0xc54e35c0 filterdns
49934     1 56772     0  S       wait     0xc5606d48 sh
46812     1 46812     0  Rs                          apinger
34158     1 34158     0  Ss      select   0xc54e3864 bsnmpd
23285     1 23285     0  Ss      select   0xc546c1a4 rtadvd
21710     1 21710  1002  Ss      select   0xc546b124 dhcpd
 8700     1  8700  1002  Ss      select   0xc50efda4 dhcpd
61147     1 60887 65534  S       select   0xc571b4e4 dnsmasq
60276     1 60276     0  Ss      kqread   0xc561de80 dhcpleases
32815 32451 32451   131  S       kqread   0xc5135d80 ospfd
32630 32451 32451   131  S       kqread   0xc54d9a80 ospfd
32451     1 32451     0  Ss      kqread   0xc54d9e80 ospfd
36594     1 36594     0  Ss      select   0xc54e34e4 ntpd
35441     1   254   123  S       select   0xc54e3c64 ntpd
47424     1 47424     0  Ss      select   0xc546cde4 openvpn
43053     1 43053     0  Ss      select   0xc571b924 openvpn
31052     1 31052     0  Ss      select   0xc571b5e4 racoon
17011 13744 17011     0  S+      ttyin    0xc4a42e70 sh
16795 13768 16795     0  S+      ttyin    0xc4a42870 sh
13768 12507 13768     0  S+      wait     0xc56c62a8 sh
13744 12170 13744     0  S+      wait     0xc5605d48 sh
13714   273 13714     0  Ss      (threaded)          sshlockout_pf
 64145                   S       nanslp   0xc1321424 sshlockout_pf
 64070                   S       piperd   0xc5150930 initial thread
12507     1 12507     0  Ss+     wait     0xc56c6aa0 login
12170     1 12170     0  Ss+     wait     0xc56c5aa0 login
 1804     1  1804     0  Ss      nanslp   0xc1321424 minicron
 1370     1  1370     0  Ss      nanslp   0xc1321424 minicron
  961     1   961     0  Ss      nanslp   0xc1321424 minicron
37905     1 37905     0  Ss      nanslp   0xc1321424 cron
37449     1 37449     0  Ss      select   0xc54e34a4 miniupnpd
37361     1 37361     0  Ss      select   0xc54e30e4 powerd
58693 56772 56772     0  S       accept   0xc55c99e6 php
58484 56772 56772     0  S       accept   0xc55c99e6 php
57805 56196 56196     0  S       accept   0xc55c984a php
57495 56196 56196     0  S       accept   0xc55c984a php
56772 55892 56772     0  Ss      wait     0xc5513000 initial thread
56196 55892 56196     0  Ss      wait     0xc5513aa0 initial thread
55892     1 55836     0  S       kqread   0xc5571680 lighttpd
31927     1 31927     0  Ss      select   0xc54e31e4 inetd
 1339     1    24     0  S+      piperd   0xc514fab8 logger
 1266     1    24     0  S+      bpf      0xc54cb900 tcpdump
  273     1   273     0  Ss      select   0xc546c9e4 syslogd
63976     1 63976     0  Rs      CPU 0               openvpn
59746     1 59746     0  Ss      select   0xc50efd64 openvpn
55035     1 55035     0  Ss      select   0xc546b164 openvpn
52931     1 52931     0  Ss      select   0xc50efa64 openvpn
20209     1 20209    65  Ss      select   0xc50ef9e4 dhclient
14772     1 14772     0  Ss      select   0xc50ef924 dhclient
11779     1 11779     0  Ss      select   0xc50ef7e4 dhcp6c
11499     1 11499     0  Ss      select   0xc50ef724 sshd
 7469     1  7469     0  Ss      (threaded)          mpd5
 64078                   S       select   0xc4bac6a4 mpd5
  267     1   267     0  Ss      select   0xc50ee164 devd
  256   254   254     0  S       kqread   0xc5135900 check_reload_status
  254     1   254     0  Ss      kqread   0xc5135c80 check_reload_status
   39     0     0     0  SL      mdwait   0xc5127800 [md0]
   23     0     0     0  SL      flowclea 0xc13351a8 [flowcleaner]
   22     0     0     0  SL      sdflush  0xc134ff20 [softdepflush]
   21     0     0     0  SL      syncer   0xc1334f94 [syncer]
   20     0     0     0  SL      vlruwt   0xc50f32a8 [vnlru]
   19     0     0     0  SL      psleep   0xc1334cc8 [bufdaemon]
   18     0     0     0  SL      pollid   0xc132097c [idlepoll]
   17     0     0     0  SL      pgzero   0xc1350bf4 [pagezero]
   16     0     0     0  SL      psleep   0xc135081c [vmdaemon]
   15     0     0     0  SL      psleep   0xc13507e4 [pagedaemon]
    9     0     0     0  SL      ccb_scan 0xc12eb654 [xpt_thrd]
    8     0     0     0  SL      pftm     0xc04f9be0 [pfpurge]
    7     0     0     0  SL      waiting_ 0xc133c418 [sctp_iterator]
   14     0     0     0  SL      (threaded)          usb
 64051                   D       -        0xc4b78d0c [usbus4]
 64050                   D       -        0xc4b78cdc [usbus4]
 64049                   D       -        0xc4b78cac [usbus4]
 64048                   D       -        0xc4b78c7c [usbus4]
 64047                   D       -        0xc4b5ddac [usbus3]
 64046                   D       -        0xc4b5dd7c [usbus3]
 64045                   D       -        0xc4b5dd4c [usbus3]
 64044                   D       -        0xc4b5dd1c [usbus3]
 64043                   D       -        0xc4b4bdac [usbus2]
 64042                   D       -        0xc4b4bd7c [usbus2]
 64041                   D       -        0xc4b4bd4c [usbus2]
 64040                   D       -        0xc4b4bd1c [usbus2]
 64039                   D       -        0xc4b37dac [usbus1]
 64038                   D       -        0xc4b37d7c [usbus1]
 64037                   D       -        0xc4b37d4c [usbus1]
 64036                   D       -        0xc4b37d1c [usbus1]
 64035                   D       -        0xc4b24dac [usbus0]
 64034                   D       -        0xc4b24d7c [usbus0]
 64033                   D       -        0xc4b24d4c [usbus0]
 64032                   D       -        0xc4b24d1c [usbus0]
    6     0     0     0  SL      crypto_r 0xc134f4cc [crypto returns]
    5     0     0     0  SL      crypto_w 0xc134f4a8 [crypto]
    4     0     0     0  SL      -        0xc131eb24 [g_down]
    3     0     0     0  SL      -        0xc131eb20 [g_up]
    2     0     0     0  SL      -        0xc131eb18 [g_event]
   13     0     0     0  SL      sleep    0xc12c1aa0 [ng_queue0]
   12     0     0     0  LL      (threaded)          intr
 64054                   I                           [irq1: atkbd0]
 64053                   I                           [irq7: ppc0]
 64052                   I                           [swi0: uart uart]
 64031                   I                           [irq21: uhci0 uhci1*]
 64030                   I                           [irq15: ata1]
 64029                   I                           [irq14: ata0]
 64028                   I                           [irq20: atapci0]
 64024                   I                           [irq9: acpi0]
 64022                   I                           [swi5: +]
 64020                   I                           [swi2: cambio]
 64016                   I                           [swi6: task queue]
 64015                   I                           [swi6: Giant taskq]
 64007                   L      *Giant    0xc4990080 [swi4: clock]
 64006                   I                           [swi3: vm]
 64005                   I                           [swi1: netisr 0]
   11     0     0     0  RL                          [idle: cpu0]
    1     0     1     0  SLs     wait     0xc498dd48 [init]
   10     0     0     0  SL      audit_wo 0xc134f840 [audit]
    0     0     0     0  SLs     (threaded)          kernel
 64027                   D       -        0xc4b15680 [em2 taskq]
 64026                   D       -        0xc4afbd80 [em1 taskq]
 64025                   D       -        0xc4aef3c0 [em0 taskq]
 64023                   D       -        0xc4a85e00 [thread taskq]
 64021                   D       -        0xc4a86000 [kqueue taskq]
 64019                   D       -        0xc4a862c0 [acpi_task_2]
 64018                   D       -        0xc4a862c0 [acpi_task_1]
 64017                   D       -        0xc4a862c0 [acpi_task_0]
 64009                   D       -        0xc4974600 [firmware taskq]
 64001                   D       sched    0xc131ec00 [swapper]

Actions #2

Updated by Seth Mos over 10 years ago

the sbappendaddr_locked() is a function that I believe comes from our one shot dumps patch which is active for our 2.0 FreeBSD 8.1 builds.

Other panics I noted have been in sbdrop(). I prepped a new firewall at work and booted it without cables attached and it rebooted in 3 minutes flat without any traffic. Dell R310, 2GB ram, AMD64, 4 igb, 2 bce.

Ideally Ermal could have a look and see if he can find fault where this comes from.

Both Apinger and a lot of other things uses sockets which this function affects. I did a quick scan for 32 bit integer limits (ipv6 being 128 bit) but couldn't easily see a buffer overflow.

Actions #3

Updated by Seth Mos about 10 years ago

  • Status changed from New to Feedback

Word is that the move to FreeBSD 9 will solve some of the issues as the ipsec socketbuffer patch will be gone.

Actions #4

Updated by Seth Mos over 9 years ago

It appears to be resolved by upgrading base to 8.3. We'll need to wait a bit more to get a definitive statement but it appears working for Jim and me.

Actions #5

Updated by Seth Mos over 9 years ago

  • Status changed from Feedback to Resolved

Considering this resolved, seen no hangs in a month

Actions

Also available in: Atom PDF