Re: [Xen-users] Nasty kernel panic

[Steven Timm <timm@xxxxxxxx>]

Thanks to the advice from all.  Some have said that I should
get new versions of e1000 and broadcom drivers for the appropriate
hardware.  It turns out that once I disabled ipv6 as best I could,
that was enough to solve the problem in question.  I rsync'ed
big squid servers and rebooted and tarred and did everything today
that I did a couple of weeks ago, and didn't have any other
nodes unexpectedly reboot.

Steve Timm


On Thu, 28 Aug 2008, Asim wrote:

> Hi,
>
> I'm also using e1000 on my domUs. I have been keeping track of e1000
> internal function calling sequences (as a part of my project). I do
> not see any ipv6 relaated calls because it is disabled. I would
> encourage you to double check whether ipv6 is actually disabled. I
> don't remember exactly but it was a little difficult actually
> disabling it.
>
> Regards,
> Asim
> On 8/28/08, Steven Timm <timm@xxxxxxxx> wrote:
> > I have seen the following kernel panic 5 times today on
> > three different machines, two of which had been stable
> > for months and one of which is a brand new install.
> >
> > We are running the x86_64 xen kernel and userland tools that came in the
> > Xen 3.1.0
> > tarball from xen.org, on top of scientific linux (redhat clone)
> > 5.1 or 5.2.
> >
> >
> > <Aug/28 12:21 pm>Unable to handle kernel NULL pointer dereference at
> > 00000000000
> > 000f4 RIP:
> > <Aug/28 12:21 pm> [<ffffffff88256375>] :ipv6:rt6_select+0x38/0x1f4
> > <Aug/28 12:21 pm>PGD 70010067 PUD 715bf067 PMD 0
> > <Aug/28 12:21 pm>Oops: 0000 [1] SMP
> > <Aug/28 12:21 pm>CPU 0
> > <Aug/28 12:21 pm>Modules linked in: dell_rbu firmware_class ipmi_devintf
> > ipmi_si
> >   ipmi_msghandler mptctl mptbase nls_utf8 nfs lockd nfs_acl xt_physdev
> > iptable_fi
> > lter ip_tables x_tables bridge ipv6 autofs4 hidp rfcomm l2cap bluetooth
> > sunrpc b
> > infmt_misc dm_multipath video thermal sbs processor i2c_ec i2c_core fan
> > containe
> > r button battery asus_acpi ac parport_pc lp parport floppy ide_cd cdrom
> > ide_flop
> > py intel_rng joydev tsdev usbkbd usbmouse piix e752x_edac edac_mc sg e1000
> > usbhi
> > d pcspkr serio_raw siimage dm_snapshot dm_zero dm_mirror dm_mod ide_disk
> > ata_pii
> > x libata megaraid_mbox sd_mod scsi_mod megaraid_mm ext3 jbd ehci_hcd
> > ohci_hcd uh
> > ci_hcd usbcore
> > <Aug/28 12:21 pm>Pid: 3075, comm: avahi-daemon Tainted: GF     2.6.18-xen
> > #1
> > <Aug/28 12:21 pm>RIP: e030:[<ffffffff88256375>]  [<ffffffff88256375>]
> > :ipv6:rt6_
> > select+0x38/0x1f4
> > <Aug/28 12:21 pm>RSP: e02b:ffffffff80526b00  EFLAGS: 00010286
> > <Aug/28 12:21 pm>RAX: ffff88006cbd6000 RBX: ffffffff88283580 RCX:
> > 00000000000000
> > 0d
> > <Aug/28 12:21 pm>RDX: 0000000000000001 RSI: 000000000000000d RDI:
> > ffff880070a3d4
> > e0
> > <Aug/28 12:21 pm>RBP: ffff880070a3d4c0 R08: ffffffff8824f148 R09:
> > ffffffff80526b
> > 60
> > <Aug/28 12:21 pm>R10: ffffffff88293906 R11: ffff880061730180 R12:
> > ffff880053e997
> > 80
> > <Aug/28 12:21 pm>R13: 0000000000000000 R14: 0000000000000003 R15:
> > 00000000000000
> > 01
> > <Aug/28 12:21 pm>FS:  00002b34a5da6370(0000) GS:ffffffff804d3000(0000)
> > knlGS:000
> > 0000000000000
> > <Aug/28 12:21 pm>CS:  e033 DS: 0000 ES: 0000
> > <Aug/28 12:21 pm>Process avahi-daemon (pid: 3075, threadinfo
> > ffff88006fc8a000, t
> > ask ffff880000b0c860)
> > <Aug/28 12:21 pm>Stack:  0000000080526bb8 00000000ffffffff
> > 0000000000000000 0000
> > 000000000000
> > <Aug/28 12:21 pm> 0000000d00000001 ffff880070a3d4e0 ffffffff8824f148
> > ffffffff882
> > 83580
> > <Aug/28 12:21 pm> ffff880070a3d4c0 ffff880053e99780 0000000000000000
> > 00000000000
> > 00003
> > <Aug/28 12:21 pm>Call Trace:
> > <Aug/28 12:21 pm> <IRQ> [<ffffffff8824f148>] :ipv6:ip6_rcv_finish+0x0/0x28
> > <Aug/28 12:21 pm> [<ffffffff882568e7>] :ipv6:ip6_route_input+0x70/0x1cf
> > <Aug/28 12:21 pm> [<ffffffff8824f3c5>] :ipv6:ipv6_rcv+0x255/0x2ba
> > <Aug/28 12:21 pm> [<ffffffff80395cbc>] netif_receive_skb+0x2d3/0x2f3
> > <Aug/28 12:21 pm> [<ffffffff8828f9b4>] :bridge:br_pass_frame_up+0x64/0x66
> > <Aug/28 12:21 pm> [<ffffffff8828fa7a>]
> > :bridge:br_handle_frame_finish+0xc4/0xf6
> > <Aug/28 12:21 pm> [<ffffffff88292e57>]
> > :bridge:br_nf_pre_routing_finish_ipv6+0xd
> > f/0xe3
> > <Aug/28 12:21 pm> [<ffffffff882935e6>]
> > :bridge:br_nf_pre_routing+0x39b/0x667
> > <Aug/28 12:21 pm> [<ffffffff803ad73c>] nf_iterate+0x52/0x79
> > <Aug/28 12:21 pm> [<ffffffff8828f9b6>]
> > :bridge:br_handle_frame_finish+0x0/0xf6
> > <Aug/28 12:21 pm> [<ffffffff803ad7d6>] nf_hook_slow+0x73/0xea
> > <Aug/28 12:21 pm> [<ffffffff8828f9b6>]
> > :bridge:br_handle_frame_finish+0x0/0xf6
> > <Aug/28 12:21 pm> [<ffffffff8828fc43>] :bridge:br_handle_frame+0x167/0x190
> > <Aug/28 12:21 pm> [<ffffffff80395c14>] netif_receive_skb+0x22b/0x2f3
> > <Aug/28 12:21 pm> [<ffffffff88107a79>]
> > :e1000:e1000_clean_rx_irq+0x430/0x4d5
> > <Aug/28 12:21 pm> [<ffffffff881074ec>] :e1000:e1000_clean+0x82/0x160
> > <Aug/28 12:21 pm> [<ffffffff80395f51>] net_rx_action+0xe7/0x254
> > <Aug/28 12:21 pm> [<ffffffff80233d97>] __do_softirq+0x7b/0x10d
> > <Aug/28 12:21 pm> [<ffffffff8020b094>] call_softirq+0x1c/0x28
> > <Aug/28 12:21 pm> [<ffffffff8020cdfd>] do_softirq+0x62/0xd9
> > <Aug/28 12:21 pm> [<ffffffff8020cc9c>] do_IRQ+0x68/0x71
> > <Aug/28 12:21 pm> [<ffffffff8034b347>] evtchn_do_upcall+0xee/0x165
> > <Aug/28 12:21 pm> [<ffffffff8020abca>] do_hypervisor_callback+0x1e/0x2c
> > <Aug/28 12:21 pm> <EOI>
> >
> > <Aug/28 12:21 pm>Code: 41 8b 85 f4 00 00 00 4d 85 ed 4d 89 ec 89 44 24 0c
> > 0f 84
> > 36
> > <Aug/28 12:21 pm>RIP  [<ffffffff88256375>] :ipv6:rt6_select+0x38/0x1f4
> > <Aug/28 12:21 pm> RSP <ffffffff80526b00>
> > <Aug/28 12:21 pm>CR2: 00000000000000f4
> > <Aug/28 12:21 pm> <0>Kernel panic - not syncing: Aiee, killing interrupt
> > handler
> >
> >
> > ------------------------------------------------
> >
> > There are different processes pid's that show as the triggering process
> > but the base error is the same.  A couple times it is triggered by the
> > swapper.
> >
> > What is puzzling is the references to ipv6 which I was pretty sure I
> > have disabled everywhere.  To be clear these crashes
> > are from the dom0, and when it happens the dom0 hangs and does
> > not auto-reboot, it requires a reset.
> >
> > Any ideas?  This config has been pretty stable for us on 7
> > different machines including these ones.  A couple of times it happened
> > just about the time we were shutting down a xen domU, a couple
> > other times today it happened on a machine that I wasn't even working on.
> >
> > Steve Timm
> >
> >
> >
> > --
> > ------------------------------------------------------------------
> > Steven C. Timm, Ph.D  (630) 840-8525
> > timm@xxxxxxxx  http://home.fnal.gov/~timm/
> > Fermilab Computing Division, Scientific Computing Facilities,
> > Grid Facilities Department, FermiGrid Services Group, Assistant Group
> > Leader.
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-users
--
------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users