WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] BUG: soft lockup

Hi all,

I've been experiencing a rash of CPU lockups on a number of domU's recently.  
It's been happening on two different servers.  About a year ago I had this 
problem every once in a while but it was not frequent.  I was running Ubuntu 
with Xen 3.1 and 2.6.24-18 back then.  I'm now running Xen 3.3 and 2.6.24-26.  

What I have noticed is that just prior to the lockups the domU's had high cpu 
loads.  The domU that I have the most problems with is a Zimbra server.  My 
guess is that a rash of spam comes through and cpu loads get high, then the 
cpu's lock up.  Originally I had it running with 1 cpu but have since upped it 
2 then 3 cpu's.  

I have been collecting the lockup messages and have posed a few below.  Any 
ideas?  Recommendations?  

Thanks,
Dana


[138077.172283]  =======================
[138075.147398] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:97]
[138075.147411] 
[138075.147419] Pid: 97, comm: kswapd0 Tainted: G      D (2.6.24-26-xen #1)
[138075.147426] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 0
[138075.147441] EIP is at _spin_lock+0x7/0x10
[138075.147447] EAX: c1da48ec EBX: 00000000 ECX: 220c7000 EDX: 00000000
[138075.147453] ESI: 8b804067 EDI: c1da48ec EBP: 00000f28 ESP: ed707dec
[138075.147459]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[138075.147471] CR0: 8005003b CR2: 080f0010 CR3: 2213b000 CR4: 00000660
[138075.147482] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[138075.147488] DR6: ffff0ff0 DR7: 00000400
[138075.147495]  [<c01773cb>] page_check_address+0x1cb/0x3c0
[138075.147514]  [<c0119868>] xen_invlpg_mask+0x38/0x40
[138075.147529]  [<c017762e>] page_referenced_one+0x6e/0x190
[138075.147541]  [<c017875c>] page_referenced+0xec/0x130
[138075.147552]  [<c01671cf>] shrink_active_list+0x18f/0x5c0
[138075.147567]  [<c016826d>] shrink_zone+0xdd/0x100
[138075.147578]  [<c01688cc>] kswapd+0x44c/0x490
[138075.147589]  [<c013bb00>] autoremove_wake_function+0x0/0x40
[138075.147603]  [<c011e270>] complete+0x40/0x60
[138075.147614]  [<c0168480>] kswapd+0x0/0x490
[138075.147625]  [<c013b842>] kthread+0x42/0x70
[138075.147635]  [<c013b800>] kthread+0x0/0x70
[138075.147646]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[138075.147658]  =======================
[138088.987826] BUG: soft lockup - CPU#1 stuck for 11s! [java:23215]
[138088.987841] 
[138088.987846] Pid: 23215, comm: java Tainted: G      D (2.6.24-26-xen #1)
[138088.987850] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 1
[138088.987862] EIP is at _spin_lock+0x7/0x10
[138088.987866] EAX: c1da48ec EBX: 00000000 ECX: c1da48e0 EDX: 00000ca8
[138088.987870] ESI: 8b804067 EDI: 00000000 EBP: e20c7ca8 ESP: e226be04
[138088.987873]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[138088.987883] CR0: 80050033 CR2: 940ef020 CR3: 2211f000 CR4: 00000660
[138088.987891] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[138088.987896] DR6: ffff0ff0 DR7: 00000400
[138088.987901]  [<c016d88d>] unmap_vmas+0x43d/0xae0
[138088.987922]  [<c011959c>] kmap_atomic+0x1c/0x30
[138088.987941]  [<c01192fd>] kunmap_atomic+0x3d/0x60
[138088.987957]  [<c0173ee8>] vma_adjust+0x1c8/0x440
[138088.987967]  [<c0173765>] unmap_region+0x95/0x120
[138088.987975]  [<c0174387>] do_munmap+0x147/0x1f0
[138088.987983]  [<c0174c90>] mmap_region+0x70/0x450
[138088.987991]  [<c01db3b7>] security_file_mmap+0x27/0x30
[138088.988001]  [<c0175472>] do_mmap_pgoff+0x312/0x330
[138088.988008]  [<c010a02b>] sys_mmap2+0xbb/0xd0
[138088.988016]  [<c0105832>] syscall_call+0x7/0xb
[138088.988023]  [<c0320000>] svc_accept+0x150/0x410
[138088.988032]  =======================


[66916.451144] BUG: soft lockup - CPU#0 stuck for 11s! [java:2758]
[66928.193453] BUG: soft lockup - CPU#1 stuck for 11s! [java:3419]


[336990.703192] BUG: soft lockup - CPU#1 stuck for 11s! [ps:32586]
[336990.703206] 
[336990.703214] Pid: 32586, comm: ps Tainted: G      D (2.6.24-26-xen #1)
[336990.703221] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 1
[336990.703235] EIP is at _spin_lock+0x7/0x10
[336990.703241] EAX: c1dbc72c EBX: 00000000 ECX: c1dbc720 EDX: 00000007
[336990.703247] ESI: 57b51067 EDI: 00000001 EBP: e2cb93c8 ESP: e2033e4c
[336990.703253]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[336990.703266] CR0: 80050033 CR2: 08079004 CR3: 23651000 CR4: 00000660
[336990.703275] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[336990.703282] DR6: ffff0ff0 DR7: 00000400
[336990.703288]  [<c0171646>] handle_mm_fault+0xae6/0x1360
[336990.703307]  [<c020e057>] rb_insert_color+0x77/0xe0
[336990.703325]  [<c032a27e>] do_page_fault+0x35e/0xe70
[336990.703337]  [<c01745d4>] vma_merge+0x144/0x1d0
[336990.703349]  [<c0174b75>] do_brk+0x195/0x240
[336990.703362]  [<c0175126>] sys_brk+0xb6/0xf0
[336990.703374]  [<c0329f20>] do_page_fault+0x0/0xe70
[336990.703384]  [<c0328bc5>] error_code+0x35/0x40
[336990.703396]  =======================
[337005.938292] BUG: soft lockup - CPU#2 stuck for 11s! [zmlocalconfig:11371]
[337005.938306] 
[337005.938312] Pid: 11371, comm: zmlocalconfig Tainted: G      D 
(2.6.24-26-xen #1)
[337005.938318] EIP: 0061:[<c03286e7>] EFLAGS: 00000286 CPU: 2
[337005.938330] EIP is at _spin_lock+0x7/0x10
[337005.938335] EAX: ec64a870 EBX: ec64a870 ECX: 00000002 EDX: ec64a871
[337005.938339] ESI: 00000000 EDI: c03fe800 EBP: c1261e38 ESP: c1261d7c
[337005.938343]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[337005.938357] CR0: 8005003b CR2: 08128000 CR3: 25d8e000 CR4: 00000660
[337005.938364] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[337005.938370] DR6: ffff0ff0 DR7: 00000400
[337005.938376]  [<c01771f0>] page_lock_anon_vma+0x20/0x30
[337005.938391]  [<c01786fd>] page_referenced+0x8d/0x130
[337005.938401]  [<c01671cf>] shrink_active_list+0x18f/0x5c0
[337005.938411]  [<c0164286>] get_dirty_limits+0x16/0x200
[337005.938421]  [<ee04b38e>] mb_cache_shrink_fn+0x1e/0x100 [mbcache]
[337005.938435]  [<c016826d>] shrink_zone+0xdd/0x100
[337005.938444]  [<c0168d72>] try_to_free_pages+0x152/0x250
[337005.938453]  [<c0162fcb>] __alloc_pages+0x14b/0x390
[337005.938463]  [<c01855c5>] do_sync_read+0xd5/0x120
[337005.938475]  [<c0163247>] __get_free_pages+0x37/0x50
[337005.938483]  [<c0124496>] copy_process+0xa6/0x1210
[337005.938493]  [<c0197c34>] d_alloc+0x114/0x1a0
[337005.938503]  [<c0125830>] do_fork+0x40/0x260
[337005.938511]  [<c0210f00>] copy_to_user+0x30/0x60
[337005.938523]  [<c0103226>] sys_clone+0x36/0x40
[337005.938530]  [<c0105832>] syscall_call+0x7/0xb
[337005.938542]  =======================
[336990.803889] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:103]
[336990.803907] 
[336990.803915] Pid: 103, comm: kswapd0 Tainted: G      D (2.6.24-26-xen #1)
[336990.803922] EIP: 0061:[<c03286ea>] EFLAGS: 00000286 CPU: 0
[336990.803940] EIP is at _spin_lock+0xa/0x10
[336990.803948] EAX: c1dbc86c EBX: 00000000 ECX: 22cc3000 EDX: 00000000
[336990.803955] ESI: 57b47067 EDI: c1dbc86c EBP: 00000ff0 ESP: ed725dec
[336990.803961]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[336990.803976] CR0: 8005003b CR2: b791e6d9 CR3: 23e3b000 CR4: 00000660
[336990.803986] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[336990.803992] DR6: ffff0ff0 DR7: 00000400
[336990.804001]  [<c01773cb>] page_check_address+0x1cb/0x3c0
[336990.804026]  [<c017762e>] page_referenced_one+0x6e/0x190
[336990.804039]  [<c017875c>] page_referenced+0xec/0x130
[336990.804049]  [<c01671cf>] shrink_active_list+0x18f/0x5c0
[336990.804064]  [<c0210556>] memmove+0x36/0x40
[336990.804079]  [<c0164286>] get_dirty_limits+0x16/0x200
[336990.804089]  [<c0139857>] call_rcu+0x97/0xa0
[336990.804102]  [<ee04b38e>] mb_cache_shrink_fn+0x1e/0x100 [mbcache]
[336990.804120]  [<c016826d>] shrink_zone+0xdd/0x100
[336990.804132]  [<c01688cc>] kswapd+0x44c/0x490
[336990.804145]  [<c013bb00>] autoremove_wake_function+0x0/0x40
[336990.804160]  [<c011e270>] complete+0x40/0x60
[336990.804172]  [<c0168480>] kswapd+0x0/0x490
[336990.804183]  [<c013b842>] kthread+0x42/0x70
[336990.804194]  [<c013b800>] kthread+0x0/0x70
[336990.804206]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[336990.804218]  =======================
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users