[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] kernel BUG at arch/x86/xen/mmu.c:1860!

  • To: Sander Eikelenboom <linux@xxxxxxxxxxxxxx>
  • From: Teck Choon Giam <giamteckchoon@xxxxxxxxx>
  • Date: Sat, 12 Mar 2011 04:29:34 +0800
  • Cc: Andreas Olsowski <andreas.olsowski@xxxxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx, benco <benco@xxxxxxx>
  • Delivery-date: Fri, 11 Mar 2011 12:30:39 -0800
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Bbqld8RB8H/PwjcqEP5M8fteEAeEmGAm7PJAdUqXR4iLb1Z6Ix3etMSJyXDp6NS0xG KqSyOTh+kKvuUgmvbuteqVT2kY0OQSOFwCjwjGSR2aeNi4Xd1foEZeYRFqVpsL4yV/r1 PyiUALtgB1EMAK/lsWsEM34QrYqzvdAgWaDBg=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>


On Sat, Mar 12, 2011 at 3:59 AM, Sander Eikelenboom <linux@xxxxxxxxxxxxxx> wrote:
Haven't seen this one on my systen. I do use LVM, but i'm only using LVM as device mapper, without software raid etc.
Also using debian lenny/squeeze together with custom compiled xen and xen-2.6.32.x pvops kernel

So I would suspect the multipathing / raid part.

From one of my test server, below is grep output:

# grep -A 45 'cut here' /var/log/messages*|grep 'kernel: Process'
/var/log/messages-Mar 10 00:53:00 xen06 kernel: Process mpath_wait (pid: 9513, threadinfo ffff880010f38000, task ffff880010c4c100)
/var/log/messages-Mar 10 02:52:33 xen06 kernel: Process clvmd (pid: 3240, threadinfo ffff880035a14000, task ffff88003b49c340)
/var/log/messages-Mar 10 02:52:33 xen06 kernel: Process syslogd (pid: 24279, threadinfo ffff88003bb3e000, task ffff880016f3a1c0)
/var/log/messages-Mar 10 03:40:52 xen06 kernel: Process mpath_wait (pid: 16487, threadinfo ffff88003becc000, task ffff88003bdb8480)
/var/log/messages.2-Feb 23 22:56:56 xen06 kernel: Process mpath_wait (pid: 15222, threadinfo ffff8800265b8000, task ffff88003b940200)
/var/log/messages.2-Feb 24 01:03:17 xen06 kernel: Process dmsetup (pid: 509, threadinfo ffff88003ced4000, task ffff88003025c4c0)
/var/log/messages.2-Feb 26 23:20:22 xen06 kernel: Process mpath_wait (pid: 10154, threadinfo ffff880025604000, task ffff8800254804c0)
/var/log/messages.4-Feb  8 04:30:10 xen06 kernel: Process mpath_wait (pid: 11646, threadinfo ffff88001e722000, task ffff880025222440)
/var/log/messages.4-Feb  9 16:43:11 xen06 kernel: Process lvremove (pid: 24469, threadinfo ffff88003b930000, task ffff880025ab03c0)
/var/log/messages.4-Feb  9 22:02:09 xen06 kernel: Process grep (pid: 10327, threadinfo ffff880025056000, task ffff880024814240)
/var/log/messages.4-Feb  9 22:13:37 xen06 kernel: Process dmsetup (pid: 6287, threadinfo ffff88003cf64000, task ffff880025fe8740)
/var/log/messages.4-Feb  9 22:16:55 xen06 kernel: Process clvmd (pid: 3204, threadinfo ffff88003ccb8000, task ffff88003a436440)

Whereby output related to 'kernel: BUG: unable to handle kernel paging request at':

# grep -B 20 -A 50 'kernel: BUG: unable to handle kernel paging request at' /var/log/messages*
/var/log/messages.2-Feb 24 00:59:13 xen06 lvm[2379]: No longer monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.2-Feb 24 00:59:14 xen06 lvm[2379]: Monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.2-Feb 24 00:59:14 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:14 xen06 kernel: EXT3 FS on dm-0, internal journal
/var/log/messages.2-Feb 24 00:59:14 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:14 xen06 lvm[2379]: No longer monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.2-Feb 24 00:59:15 xen06 lvm[2379]: Monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.2-Feb 24 00:59:16 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:16 xen06 kernel: EXT3 FS on dm-0, internal journal
/var/log/messages.2-Feb 24 00:59:16 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:16 xen06 lvm[2379]: No longer monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.2-Feb 24 00:59:17 xen06 lvm[2379]: Monitoring snapshot XenGroup-testcrash3--snapshot
/var/log/messages.2-Feb 24 00:59:17 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:17 xen06 kernel: EXT3 FS on dm-0, internal journal
/var/log/messages.2-Feb 24 00:59:17 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:18 xen06 lvm[2379]: No longer monitoring snapshot XenGroup-testcrash3--snapshot
/var/log/messages.2-Feb 24 00:59:19 xen06 lvm[2379]: Monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: EXT3 FS on dm-0, internal journal
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.2:Feb 24 00:59:19 xen06 kernel: BUG: unable to handle kernel paging request at ffff88002561f3f8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: IP: [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: PGD 1002067 PUD 1006067 PMD 1f1067 PTE 801000002561f065
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Oops: 0003 [#1] SMP
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: last sysfs file: /sys/block/dm-13/dev
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CPU 1
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Modules linked in: dlm configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg ide_cd_mod cdrom tg3 serio_raw libphy button tpm_tis tpm tpm_bios pcspkr shpchp i2c_i801 i2c_core iTCO_wdt dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Pid: 16520, comm: udevd Not tainted 2.6.32.29-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RIP: e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RSP: e02b:ffff880025655b88  EFLAGS: 00010246
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RAX: 0000000000000000 RBX: ffff88002561f3f8 RCX: 00007f9a10c63000
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RDX: ffffea0000000000 RSI: 0000000000000000 RDI: ffff88002561f3f8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RBP: ffff880025655b98 R08: 00007f9a10fc0000 R09: ffff88003e64d609
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: R10: ffff88003cce22c0 R11: ffffffff813391a0 R12: 0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 lvm[2379]: No longer monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: R13: 00007f9a10000000 R14: ffff88003cc6a7f8 R15: ffff88002550c340
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: FS:  00007f9a10a41710(0000) GS:ffff88002806c000(0000) knlGS:0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 udevd-event[16460]: run_program: '/sbin/dmsetup' abnormal exit
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CR2: ffff88002561f3f8 CR3: 000000003bb46000 CR4: 0000000000002660
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Process udevd (pid: 16520, threadinfo ffff880025654000, task ffff88003cce22c0)
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Stack:
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  ffff88002561f3f8 0000000025690067 ffff880025655c48 ffffffff810a60b0
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: <0> ffff880025655ac8 000000000000001b 00007f9a10e00000 0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: <0> 00007f9a10c63000 ffff88002807a5c0 00007f9a10c63000 ffff880025655cb8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Call Trace:
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a60b0>] free_pgd_range+0x3b3/0x401
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a63b1>] free_pgtables+0xb6/0xce
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a84a4>] exit_mmap+0xf0/0x14b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff81048888>] mmput+0x46/0xb2
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c8cd9>] flush_old_exec+0x463/0x536
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810fbf99>] load_elf_binary+0x365/0x171b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810faa30>] ? load_misc_binary+0x61/0x322
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a50dd>] ? get_user_pages+0x44/0x46
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c795e>] ? put_arg_page+0x9/0xb
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c7e0b>] search_binary_handler+0xde/0x268
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c9208>] do_execve+0x193/0x262
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810113af>] sys_execve+0x3e/0x58
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff81012f1a>] stub_execve+0x6a/0xc0
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Code: 8b 3c 25 70 c5 00 00 ff 14 25 68 a2 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48 89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff ff ff 5b 41 5c c9 c3
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RIP  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  RSP <ffff880025655b88>
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CR2: ffff88002561f3f8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: ---[ end trace 5ea31e622470b518 ]---
/var/log/messages.2-Feb 24 00:59:20 xen06 lvm[2379]: Monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.2-Feb 24 00:59:21 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:21 xen06 kernel: EXT3 FS on dm-0, internal journal
/var/log/messages.2-Feb 24 00:59:21 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:21 xen06 lvm[2379]: No longer monitoring snapshot XenGroup-testcrash5--snapshot
--
/var/log/messages.4-Feb  8 03:58:30 xen06 lvm[3961]: Monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  8 03:58:30 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:30 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  8 03:58:30 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:30 xen06 lvm[3961]: No longer monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  8 03:58:31 xen06 lvm[3961]: Monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  8 03:58:31 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:31 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  8 03:58:31 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:32 xen06 lvm[3961]: No longer monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  8 03:58:33 xen06 lvm[3961]: Monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  8 03:58:33 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:33 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  8 03:58:33 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:33 xen06 lvm[3961]: No longer monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  8 03:58:34 xen06 lvm[3961]: Monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:34 xen06 lvm[3961]: No longer monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4:Feb  8 03:58:34 xen06 kernel: BUG: unable to handle kernel paging request at ffff880030081010
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: IP: [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: PGD 1002067 PUD 1006067 PMD 246067 PTE 8010000030081065
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Oops: 0003 [#1] SMP
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CPU 3
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp gfs2 dlm configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg ide_cd_mod cdrom serio_raw tg3 button libphy tpm_tis tpm tpm_bios i2c_i801 pcspkr i2c_core iTCO_wdt shpchp dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Pid: 32051, comm: mpath_wait Not tainted 2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RIP: e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RSP: e02b:ffff880030c19b88  EFLAGS: 00010246
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RAX: 0000000000000000 RBX: ffff880030081010 RCX: 00000000008c3000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RDX: ffffea0000000000 RSI: 0000000000000000 RDI: ffff880030081010
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RBP: ffff880030c19b98 R08: 0000000002150000 R09: ffff88003e6f3e89
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: R10: ffff880030e78040 R11: ffffffff813381a0 R12: 0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: R13: 0000000000600000 R14: ffff880036dc5000 R15: ffff880030994000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: FS:  00007f147d9606e0(0000) GS:ffff8800280a6000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CR2: ffff880030081010 CR3: 000000003ba55000 CR4: 0000000000002660
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Process mpath_wait (pid: 32051, threadinfo ffff880030c18000, task ffff88003c93c280)
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Stack:
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  ffff880030081010 000000003b7e4067 ffff880030c19c48 ffffffff810a5e32
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: <0> ffff880030c19ac8 000000000000001f 0000000002000000 0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: <0> 00000000008c3000 ffff8800280b45c0 00000000008c3000 ffff880030c19cb8
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a5e32>] free_pgd_range+0x3b3/0x401
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a6133>] free_pgtables+0xb6/0xce
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a8224>] exit_mmap+0xf0/0x14b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff81048648>] mmput+0x46/0xb2
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c8a56>] flush_old_exec+0x460/0x533
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810fbd15>] load_elf_binary+0x365/0x171c
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810fa7ac>] ? load_misc_binary+0x61/0x322
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a4e5f>] ? get_user_pages+0x44/0x46
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c76de>] ? put_arg_page+0x9/0xb
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c7b8b>] search_binary_handler+0xde/0x268
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c8f85>] do_execve+0x193/0x262
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810113af>] sys_execve+0x3e/0x58
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff81012f1a>] stub_execve+0x6a/0xc0
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Code: 8b 3c 25 70 c5 00 00 ff 14 25 68 82 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48 89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff ff ff 5b 41 5c c9 c3
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RIP  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  RSP <ffff880030c19b88>
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CR2: ffff880030081010
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: ---[ end trace f2d7fcc13c4cdf6a ]---
/var/log/messages.4:Feb  8 03:58:34 xen06 kernel: BUG: unable to handle kernel paging request at ffff8800313e1b00
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: IP: [<ffffffff810377f2>] ptep_set_access_flags+0x27/0x4a
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: PGD 1002067 PUD 1006067 PMD 24f067 PTE 80100000313e1065
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Oops: 0003 [#2] SMP
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CPU 1
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp gfs2 dlm configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg ide_cd_mod cdrom serio_raw tg3 button libphy tpm_tis tpm tpm_bios i2c_i801 pcspkr i2c_core iTCO_wdt shpchp dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Pid: 32053, comm: mpath_wait Tainted: G      D    2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RIP: e030:[<ffffffff810377f2>]  [<ffffffff810377f2>] ptep_set_access_flags+0x27/0x4a
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RSP: e02b:ffff8800307c7cb8  EFLAGS: 00210202
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RAX: ffff8800304b4870 RBX: 00007f147d960770 RCX: 80000001f6d6f167
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RDX: ffff8800313e1b00 RSI: 00007f147d960770 RDI: ffff8800304b4870
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RBP: ffff8800307c7cd8 R08: 0000000000000001 R09: ffffea0000ac5948
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: R10: 3138666666666666 R11: 000000000002496a R12: 0000000000000001
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: R13: ffff8800304b4870 R14: 0000000000000008 R15: 0000000000000000
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: FS:  00007f147d9606e0(0000) GS:ffff88002806c000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CR2: ffff8800313e1b00 CR3: 0000000031261000 CR4: 0000000000002660
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Process mpath_wait (pid: 32053, threadinfo ffff8800307c6000, task ffff880030ece200)
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Stack:
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  80000001f6d6f145 0000000000000000 ffffea0000ac5948 ffffea0000800f30
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: <0> ffff8800307c7d88 ffffffff810a2a19 0000000000000001 ffff8800307c7de8
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: <0> ffffffff8100f0b1 ffff8800307c7d98 ffff8800305a8f60 ffff8800313e1b00
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff810a2a19>] do_wp_page+0x2fd/0x717
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100f0b1>] ? xen_force_evtchn_callback+0xd/0xf
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100cde0>] ? xen_pte_val+0x64/0x6e
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100c3e9>] ? __raw_callee_save_xen_pte_val+0x15/0x23
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff810a4933>] handle_mm_fault+0x84a/0x8b9
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8131be2d>] do_page_fault+0x252/0x2e2
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100f7f2>] ? check_events+0x12/0x20
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff81319db5>] page_fault+0x25/0x30
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8116dc9d>] ? __put_user_4+0x1d/0x30
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff81047d2c>] ? schedule_tail+0x98/0x9d
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff810129c3>] ret_from_fork+0x13/0x80
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Code: 41 5d c9 c3 55 48 89 e5 41 55 49 89 fd 41 54 45 31 e4 53 48 89 f3 48 83 ec 08 48 39 0a 41 0f 95 c4 45 85 e4 74 1d 45 85 c0 74 18 <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 48 89 de 4c 89 ef e8 37
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RIP  [<ffffffff810377f2>] ptep_set_access_flags+0x27/0x4a
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  RSP <ffff8800307c7cb8>
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CR2: ffff8800313e1b00
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: ---[ end trace f2d7fcc13c4cdf6b ]---
/var/log/messages.4-Feb  8 04:26:35 xen06 syslogd 1.4.1: restart.
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: klogd 1.4.1, log source = /proc/kmsg started.
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: Linux version 2.6.32.28-0.xen.pvops.choon.centos5 (mockbuild@xxxxxxxxxxxxxxxxxx) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Feb 8 03:26:17 SGT 2011
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: Command line: ro root=/dev/md1 panic=5 panic_timeout=5
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: KERNEL supported cpus:
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel:   Intel GenuineIntel
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel:   AMD AuthenticAMD
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel:   Centaur CentaurHauls
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: released 0 pages of unused memory
--
/var/log/messages.4-Feb  9 07:44:11 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 07:44:11 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:11 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 07:44:11 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:11 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 07:44:12 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 07:44:12 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:12 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 07:44:12 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:12 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 07:44:13 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  9 07:44:13 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:13 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 07:44:13 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:13 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  9 07:44:14 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  9 07:44:14 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:15 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 07:44:15 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:15 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4:Feb  9 07:53:59 xen06 kernel: BUG: unable to handle kernel paging request at ffff88003cc0c010
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: IP: [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: PGD 1002067 PUD 1006067 PMD 2ac067 PTE 801000003cc0c065
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Oops: 0003 [#1] SMP
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CPU 0
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Modules linked in: dlm configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg ide_cd_mod cdrom serio_raw tg3 libphy button tpm_tis tpm tpm_bios pcspkr i2c_i801 shpchp i2c_core iTCO_wdt dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Pid: 11005, comm: mpath_wait Not tainted 2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RIP: e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RSP: e02b:ffff88003b7d3b88  EFLAGS: 00010246
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RAX: 0000000000000000 RBX: ffff88003cc0c010 RCX: 00000000008c3000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RDX: ffffea0000000000 RSI: 0000000000000000 RDI: ffff88003cc0c010
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RBP: ffff88003b7d3b98 R08: 0000000001244000 R09: ffff88003f009f09
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: R10: ffff8800269e3ac0 R11: ffffffff813381a0 R12: 0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: R13: 0000000000600000 R14: ffff8800269ca000 R15: ffff8800268bb000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: FS:  00007fa7f8de76e0(0000) GS:ffff88002804f000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CR2: ffff88003cc0c010 CR3: 00000000306b0000 CR4: 0000000000002660
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Process mpath_wait (pid: 11005, threadinfo ffff88003b7d2000, task ffff8800307aa480)
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Stack:
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  ffff88003cc0c010 000000003cfee067 ffff88003b7d3c48 ffffffff810a5e32
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: <0> 000000000000e02b 0000000000000060 0000000001200000 0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: <0> 00000000008c3000 ffff88002805d5c0 00000000008c3000 ffff88003b7d3cb8
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a5e32>] free_pgd_range+0x3b3/0x401
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a6133>] free_pgtables+0xb6/0xce
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a8224>] exit_mmap+0xf0/0x14b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff81048648>] mmput+0x46/0xb2
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c8a56>] flush_old_exec+0x460/0x533
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810fbd15>] load_elf_binary+0x365/0x171c
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810fa7ac>] ? load_misc_binary+0x61/0x322
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a4e5f>] ? get_user_pages+0x44/0x46
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c76de>] ? put_arg_page+0x9/0xb
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c7b8b>] search_binary_handler+0xde/0x268
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c8f85>] do_execve+0x193/0x262
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810113af>] sys_execve+0x3e/0x58
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff81012f1a>] stub_execve+0x6a/0xc0
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Code: 8b 3c 25 70 c5 00 00 ff 14 25 68 82 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48 89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff ff ff 5b 41 5c c9 c3
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RIP  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  RSP <ffff88003b7d3b88>
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CR2: ffff88003cc0c010
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: ---[ end trace b5fffc59f6c8f974 ]---
/var/log/messages.4-Feb  9 16:31:21 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:31:21 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:31:21 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 16:31:21 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:31:21 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:31:22 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 16:31:22 xen06 kernel: kjournald starting.  Commit interval 5 seconds
--
/var/log/messages.4-Feb  9 16:35:34 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:35:34 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:34 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 16:35:34 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:35 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:35:36 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 16:35:36 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:36 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 16:35:36 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:36 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 16:35:37 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 16:35:37 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:37 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 16:35:37 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:37 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 16:35:38 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  9 16:35:38 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:39 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4:Feb  9 16:35:39 xen06 kernel: BUG: unable to handle kernel paging request at ffff880026b00418
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: IP: [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: PGD 1002067 PUD 1006067 PMD 1fb067 PTE 8010000026b00065
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Oops: 0003 [#2] SMP
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CPU 2
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Modules linked in: dlm configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg ide_cd_mod cdrom serio_raw tg3 libphy button tpm_tis tpm tpm_bios pcspkr i2c_i801 shpchp i2c_core iTCO_wdt dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Pid: 25953, comm: udevd Tainted: G      D    2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RIP: e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RSP: e02b:ffff88002682fb88  EFLAGS: 00010246
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RAX: 0000000000000000 RBX: ffff880026b00418 RCX: 00007f71114ac000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RDX: ffffea0000000000 RSI: 0000000000000000 RDI: ffff880026b00418
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RBP: ffff88002682fb98 R08: 00007f7112d14000 R09: ffff88003e6e5a89
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: R10: ffff88003d2d0580 R11: ffffffff813381a0 R12: 0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: R13: 00007f7110800000 R14: ffff880026b047f0 R15: ffff880025943e20
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: FS:  00007f711128a710(0000) GS:ffff880028089000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CR2: ffff880026b00418 CR3: 0000000026946000 CR4: 0000000000002660
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Process udevd (pid: 25953, threadinfo ffff88002682e000, task ffff8800269ae380)
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Stack:
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  ffff880026b00418 000000003d32d067 ffff88002682fc48 ffffffff810a5e32
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: <0> ffff88002682fac8 000000000000001c 00007f7112c00000 0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 udevd-event[25941]: run_program: '/sbin/dmsetup' abnormal exit
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: <0> 00007f71114ac000 ffff8800280975c0 00007f71114ac000 ffff88002682fcb8
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a5e32>] free_pgd_range+0x3b3/0x401
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a6133>] free_pgtables+0xb6/0xce
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a8224>] exit_mmap+0xf0/0x14b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff81048648>] mmput+0x46/0xb2
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c8a56>] flush_old_exec+0x460/0x533
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810fbd15>] load_elf_binary+0x365/0x171c
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810fa7ac>] ? load_misc_binary+0x61/0x322
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a4e5f>] ? get_user_pages+0x44/0x46
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c76de>] ? put_arg_page+0x9/0xb
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c7b8b>] search_binary_handler+0xde/0x268
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c8f85>] do_execve+0x193/0x262
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810113af>] sys_execve+0x3e/0x58
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff81012f1a>] stub_execve+0x6a/0xc0
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Code: 8b 3c 25 70 c5 00 00 ff 14 25 68 82 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48 89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff ff ff 5b 41 5c c9 c3
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RIP  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  RSP <ffff88002682fb88>
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CR2: ffff880026b00418
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: ---[ end trace b5fffc59f6c8f975 ]---
/var/log/messages.4-Feb  9 16:35:40 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  9 16:35:40 xen06 kernel: kjournald starting.  Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:40 xen06 kernel: EXT3 FS on dm-11, internal journal
/var/log/messages.4-Feb  9 16:35:40 xen06 kernel: EXT3-fs: mounted filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:40 xen06 lvm[3802]: No longer monitoring snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  9 16:35:41 xen06 lvm[3802]: Monitoring snapshot XenGroup-testcrash1--snapshot

Thanks.

Kindest regards,
Giam Teck Choon
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.