WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] BUG at xen4.1/kernel 2.6.32.35 at a CentOS 5.5 when star

To: Gerd Jakobovitsch <gerd@xxxxxxxxxxx>
Subject: Re: [Xen-devel] BUG at xen4.1/kernel 2.6.32.35 at a CentOS 5.5 when starting a VM
From: Teck Choon Giam <giamteckchoon@xxxxxxxxx>
Date: Thu, 31 Mar 2011 10:44:31 +0800
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Wed, 30 Mar 2011 19:45:34 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Zr2SLFVwzDnFCYwWGXL0uSuZGhkWxgTDQjEmOAWH5NQ=; b=SWmykQ4kUFI66IcOfqcaV4dUbM30PSOOVLYUfkDL0g8jeyLBd+4IYJNrZXse/dNtmd ROyJi5tQf4JBVEelYulc0zu0iBW65/frxuFlEUW7z3gRCqRBEcREoKMZlxzuuyqAbk4M va8ku89me+94Q+EKJNIiwnlGaP7RaxZ7cJjNI=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=VsgeN56bfuiRGKwKx2N/DKxAUQuyJvBPZQ1UA3pC4K442gEf+v0cvLdmklPhFmuBqd A8kLlVFLNDN/sHdRZLkkffPrGlbYEbePr/kGdtVj//xPXZLHyhsG5QBt/eaWsXrlxJHQ yRhKmSTE4W1ttl5nz3oZrgkenS9rIN9JyyYJM=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4D933ADB.8060106@xxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4D933ADB.8060106@xxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Wed, Mar 30, 2011 at 10:14 PM, Gerd Jakobovitsch <gerd@xxxxxxxxxxx> wrote:
>
> Hello all,
>
> I used to run xen4.0 kernel 2.6.32.24 over CentOS 5.5, with a relative 
> success, but the bug at mmu.c appeared once at a while. Therefore, I'm 
> looking for a more stable option.
> I compiled and ran the newly released xen 4.1, with kernel PVOPS 2.6.32.35 
> over CentOS 5.5. When trying to start a VM, the following bugs appeared at 
> dmesg. After that, xl and xm commands do not longer respond:
>
> [  145.749573]   alloc irq_desc for 2209 on node -1
> [  145.749581]   alloc kstat_irqs on node -1
> [  145.883515] block tda: sector-size: 512 capacity: 262144
> [  145.889952] general protection fault: 0000 [#1] SMP
> [  145.890109] last sysfs file: /sys/block/tda/removable
> [  145.890164] CPU 7
> [  145.890252] Modules linked in: bridge stp nfs fscache nfs_acl auth_rpcgss 
> arptable_filter arp_tables xt_esp ipt_ah xt_physdev xt_multiport lockd sunrpc 
> bonding dm_multipath bnx2 megaraid_sas
> [  145.891125] Pid: 5179, comm: tapdisk2 Not tainted 2.6.32.35 #1 PowerEdge 
> M610
> [  145.891184] RIP: e030:[<ffffffff81281e79>]  [<ffffffff81281e79>] 
> blktap_device_end_request+0x4e/0x63
> [  145.891296] RSP: e02b:ffff880064061cd8  EFLAGS: 00010046
> [  145.891351] RAX: 6b6b6b6b6b6b6b6b RBX: ffff88007d264690 RCX: 
> 0000000000000028
> [  145.891410] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 
> 0000000000000000
> [  145.891469] RBP: ffff880064061cf8 R08: 0000000064061c98 R09: 
> ffff88007da42948
> [  145.891528] R10: ffffea0000000008 R11: 0000000001f60080 R12: 
> ffff88007da427f8
> [  145.891587] R13: ffff88007c75f398 R14: 0000000000000000 R15: 
> ffff88007c75f3a8
> [  145.891651] FS:  00007ff33d9a4730(0000) GS:ffff8800189e5000(0000) 
> knlGS:0000000000000000
> [  145.891714] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  145.891771] CR2: 0000000002594cc8 CR3: 000000007be61000 CR4: 
> 0000000000002660
> [  145.891830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [  145.891890] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> 0000000000000400
> [  145.892171] Process tapdisk2 (pid: 5179, threadinfo ffff880064060000, task 
> ffff88007c272d60)
> [  145.892669] Stack:
> [  145.892934]  ffff88007c272d60 0000000000000000 0000000000000000 
> 0000000000000000
> [  145.893334] <0> ffff880064061e88 ffffffff812815ae ffff880064061e58 
> ffffffff811d234f
> [  145.894035] <0> ffff88007e9bbfc0 ffff88007c75f398 00000001ffffffff 
> 0000000000000000
> [  145.895015] Call Trace:
> [  145.895286]  [<ffffffff812815ae>] blktap_ring_ioctl+0x183/0x2d8
> [  145.895566]  [<ffffffff811d234f>] ? inode_has_perm+0x77/0x89
> [  145.895844]  [<ffffffff811d234f>] ? inode_has_perm+0x77/0x89
> [  145.896124]  [<ffffffff81219e24>] ? _raw_spin_lock+0x77/0x12f
> [  145.896403]  [<ffffffff81219d28>] ? _raw_spin_unlock+0xab/0xb2
> [  145.896682]  [<ffffffff81529311>] ? _spin_unlock+0x9/0xb
> [  145.896958]  [<ffffffff81219e24>] ? _raw_spin_lock+0x77/0x12f
> [  145.897234]  [<ffffffff811d2415>] ? file_has_perm+0xb4/0xc6
> [  145.897513]  [<ffffffff810fe868>] vfs_ioctl+0x5e/0x77
> [  145.897786]  [<ffffffff810fed7d>] do_vfs_ioctl+0x484/0x4d5
> [  145.898060]  [<ffffffff810fee25>] sys_ioctl+0x57/0x7a
> [  145.898338]  [<ffffffff81013d02>] system_call_fastpath+0x16/0x1b
> [  145.898614] Code: e8 5f f4 ff ff 49 8b 44 24 40 48 8b b8 80 03 00 00 e8 64 
> 75 2a 00 41 8b 54 24 60 44 89 f6 4c 89 e7 e8 b5 89 f7 ff 49 8b 44 24 40 <48> 
> 8b b8 80 03 00 00 e8 23 74 2a 00 5b 41 5c 41 5d 41 5e c9 c3
> [  145.902008] RIP  [<ffffffff81281e79>] blktap_device_end_request+0x4e/0x63
> [  145.902321]  RSP <ffff880064061cd8>
> [  145.902585] ---[ end trace 2800cfa5aa85ca0a ]---
> [  262.100689] BUG: spinlock lockup on CPU#4, vol_id/5181, ffff88007c75f520
> [  262.100965] Pid: 5181, comm: vol_id Tainted: G      D    2.6.32.35 #1
> [  262.101232] Call Trace:
> [  262.101497]  [<ffffffff81219eae>] _raw_spin_lock+0x101/0x12f
> [  262.101762]  [<ffffffff815293e6>] _spin_lock_irq+0x1e/0x20
> [  262.102028]  [<ffffffff811fcc14>] __make_request+0x5e/0x402
> [  262.102294]  [<ffffffff8101019f>] ? xen_restore_fl_direct_end+0x0/0x1
> [  262.102563]  [<ffffffff811fa171>] generic_make_request+0x258/0x2f4
> [  262.102832]  [<ffffffff811156d8>] ? bio_init+0x18/0x32
> [  262.103099]  [<ffffffff811fbec8>] submit_bio+0xd0/0xd9
> [  262.103366]  [<ffffffff81111574>] submit_bh+0xf7/0x11a
> [  262.103631]  [<ffffffff8111448f>] block_read_full_page+0x246/0x264
> [  262.103898]  [<ffffffff81117c13>] ? blkdev_get_block+0x0/0x4d
> [  262.104165]  [<ffffffff815292c6>] ? _spin_unlock_irq+0x1e/0x20
> [  262.104433]  [<ffffffff810ba73d>] ? add_to_page_cache_locked+0xa0/0xca
> [  262.104702]  [<ffffffff81116ef9>] blkdev_readpage+0x13/0x15
> [  262.104972]  [<ffffffff810c1d36>] __do_page_cache_readahead+0x144/0x177
> [  262.105240]  [<ffffffff810c1f8f>] ondemand_readahead+0x126/0x18e
> [  262.105507]  [<ffffffff810c20d7>] page_cache_sync_readahead+0x38/0x3a
> [  262.105778]  [<ffffffff810bb833>] generic_file_aio_read+0x24c/0x5c1
> [  262.106045]  [<ffffffff810f1808>] do_sync_read+0xe2/0x126
> [  262.106315]  [<ffffffff81068a02>] ? autoremove_wake_function+0x0/0x38
> [  262.106584]  [<ffffffff811d701c>] ? selinux_file_permission+0x5c/0x10e
> [  262.106854]  [<ffffffff811ce9c4>] ? security_file_permission+0x11/0x13
> [  262.107120]  [<ffffffff810f1f7b>] vfs_read+0xab/0x167
> [  262.107385]  [<ffffffff810f2374>] sys_read+0x47/0x70
> [  262.107652]  [<ffffffff81013d02>] system_call_fastpath+0x16/0x1b
> [  262.107918] sending NMI to all CPUs:
> [  262.108189] BUG: unable to handle kernel paging request at ffffffffff5fb310
> [  262.108526] IP: [<ffffffff8102c7d1>] flat_send_IPI_mask+0x6a/0xc0
> [  262.108832] PGD 1003067 PUD 1004067 PMD 18b7067 PTE 0
> [  262.109235] Oops: 0002 [#2] SMP
> [  262.109565] last sysfs file: /sys/class/blktap2/blktap1/dev
> [  262.109830] CPU 4
> [  262.110121] Modules linked in: bridge stp nfs fscache nfs_acl auth_rpcgss 
> arptable_filter arp_tables xt_esp ipt_ah xt_physdev xt_multiport lockd sunrpc 
> bonding dm_multipath bnx2 megaraid_sas
> [  262.111520] Pid: 5181, comm: vol_id Tainted: G      D    2.6.32.35 #1 
> PowerEdge M610
> [  262.112008] RIP: e030:[<ffffffff8102c7d1>]  [<ffffffff8102c7d1>] 
> flat_send_IPI_mask+0x6a/0xc0
> [  262.112535] RSP: e02b:ffff88006778f968  EFLAGS: 00010086
> [  262.112800] RAX: 00000000ff000000 RBX: ffffffff81790060 RCX: 
> 00000000000160a0
> [  262.113068] RDX: ffff88001898e000 RSI: 0000000000000002 RDI: 
> ffffffff81816020
> [  262.113337] RBP: ffff88006778f988 R08: 0000000000000000 R09: 
> 0000000000000004
> [  262.113605] R10: 0000000000000002 R11: 0000000000000004 R12: 
> 0000000000000002
> [  262.113877] R13: 0000000000000800 R14: 00000000000000ff R15: 
> 0000000000000000
> [  262.114149] FS:  00007fa78bcc5710(0063) GS:ffff88001898e000(0000) 
> knlGS:0000000000000000
> [  262.114636] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  262.114902] CR2: ffffffffff5fb310 CR3: 00000000641b4000 CR4: 
> 0000000000002660
> [  262.115171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [  262.115438] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> 0000000000000400
> [  262.115707] Process vol_id (pid: 5181, threadinfo ffff88006778e000, task 
> ffff88007db86250)
> [  262.116194] Stack:
> [  262.116451]  0000000000000000 0000000076e9ecd0 0000000000000000 
> 0000000076e9ecd0
> [  262.116825] <0> ffff88006778f998 ffffffff8102c841 ffff88006778f9b8 
> ffffffff81029f0d
> [  262.117485] <0> ffff88007c75f520 ffff88007c75f520 ffff88006778f9f8 
> ffffffff81219eb3
> [  262.118396] Call Trace:
> [  262.118657]  [<ffffffff8102c841>] flat_send_IPI_all+0x1a/0x56
> [  262.118925]  [<ffffffff81029f0d>] arch_trigger_all_cpu_backtrace+0x45/0x66
> [  262.119195]  [<ffffffff81219eb3>] _raw_spin_lock+0x106/0x12f
> [  262.119463]  [<ffffffff815293e6>] _spin_lock_irq+0x1e/0x20
> [  262.119730]  [<ffffffff811fcc14>] __make_request+0x5e/0x402
> [  262.119996]  [<ffffffff8101019f>] ? xen_restore_fl_direct_end+0x0/0x1
> [  262.120264]  [<ffffffff811fa171>] generic_make_request+0x258/0x2f4
> [  262.120532]  [<ffffffff811156d8>] ? bio_init+0x18/0x32
> [  262.120799]  [<ffffffff811fbec8>] submit_bio+0xd0/0xd9
> [  262.121066]  [<ffffffff81111574>] submit_bh+0xf7/0x11a
> [  262.121333]  [<ffffffff8111448f>] block_read_full_page+0x246/0x264
> [  262.121602]  [<ffffffff81117c13>] ? blkdev_get_block+0x0/0x4d
> [  262.121870]  [<ffffffff815292c6>] ? _spin_unlock_irq+0x1e/0x20
> [  262.122137]  [<ffffffff810ba73d>] ? add_to_page_cache_locked+0xa0/0xca
> [  262.127766]  [<ffffffff81116ef9>] blkdev_readpage+0x13/0x15
> [  262.128025]  [<ffffffff810c1d36>] __do_page_cache_readahead+0x144/0x177
> [  262.128288]  [<ffffffff810c1f8f>] ondemand_readahead+0x126/0x18e
> [  262.128548]  [<ffffffff810c20d7>] page_cache_sync_readahead+0x38/0x3a
> [  262.128810]  [<ffffffff810bb833>] generic_file_aio_read+0x24c/0x5c1
> [  262.129070]  [<ffffffff810f1808>] do_sync_read+0xe2/0x126
> [  262.129329]  [<ffffffff81068a02>] ? autoremove_wake_function+0x0/0x38
> [  262.129590]  [<ffffffff811d701c>] ? selinux_file_permission+0x5c/0x10e
> [  262.129851]  [<ffffffff811ce9c4>] ? security_file_permission+0x11/0x13
> [  262.130110]  [<ffffffff810f1f7b>] vfs_read+0xab/0x167
> [  262.130368]  [<ffffffff810f2374>] sys_read+0x47/0x70
> [  262.130624]  [<ffffffff81013d02>] system_call_fastpath+0x16/0x1b
> [  262.130883] Code: 8b 05 b4 95 7e 00 83 fe 02 44 8b 68 34 75 0a ff 90 58 01 
> 00 00 eb 0e f3 90 8b 04 25 00 b3 5f ff f6 c4 10 75 f2 44 89 f0 c1 e0 18 <89> 
> 04 25 10 b3 5f ff 41 83 fc 02 74 08 44 89 e0 44 09 e8 eb 06
> [  262.133866] RIP  [<ffffffff8102c7d1>] flat_send_IPI_mask+0x6a/0xc0
> [  262.134164]  RSP <ffff88006778f968>
> [  262.134419] CR2: ffffffffff5fb310
> [  262.134673] ---[ end trace 2800cfa5aa85ca0b ]---
>

Can you try to recompile your PVOPS kernel with CONFIG_DEBUG_PAGEALLOC=y?

You can read more about this BUG at
http://lists.xensource.com/archives/html/xen-devel/2011-03/msg01756.html

I initially hit this BUG sometime Dec 2010... ...
http://lists.xensource.com/archives/html/xen-devel/2010-12/msg01501.html

Thanks.

Kindest regards,
Giam Teck Choon

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>