[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] 2.6.32 domU getting stuck



This is on a Debian 2.6.32 lenny-backports amd64 domU - SysRq+W (show
blocked tasks) says:

[...]
[78137.427846] apache2       D 0000000000000000     0  9566  24433 0x00000000
[78137.427846]  ffff8800bfbcd4c0 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff88001889ffd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001a662e0 ffff880001a665d8 
0000000200000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034f4780     0  9568  24433 0x00000000
[78137.427846]  ffff8800bddf7810 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880035f2dfd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001a67100 ffff880001a673f8 
0000000500000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff81097d1c>] ? __call_rcu+0x110/0x118
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff81152a9f>] ? cap_file_free_security+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034b8780     0  9570  24433 0x00000000
[78137.427846]  ffff880002a50710 0000000000000286 ffff880053f9db38 
ffff880053f9db34
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880053f9dfd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001a60000 ffff880001a602f8 
0000000300000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846]  [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000345e780     0  9572  24433 0x00000000
[78137.427846]  ffff880083928710 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800a7819fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880002abf810 ffff880002abfb08 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff81097d1c>] ? __call_rcu+0x110/0x118
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff81152a9f>] ? cap_file_free_security+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff880003512780     0  9574  24433 0x00000000
[78137.427846]  ffff8800865fa350 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800094d5fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880002abc6a0 ffff880002abc998 
0000000600000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff811099c9>] ? splice_from_pipe+0x58/0x63
[78137.427846]  [<ffffffff810ff546>] ? touch_atime+0x7c/0x127
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000347c780     0  9576  24433 0x00000000
[78137.427846]  ffff880092e862e0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880051e4ffd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880002ab8710 ffff880002ab8a08 
0000000100000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000345e780     0  9577  24433 0x00000000
[78137.427846]  ffff8800018aa350 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880021df1fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880002aba350 ffff880002aba648 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff8100e5d4>] ? xen_smp_send_reschedule+0x0/0x7
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846]  [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000345e780     0  9588  24433 0x00000000
[78137.427846]  ffff880086d82350 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff88003407dfd8 
0000000000015780
[78137.427846]  0000000000015780 ffff8800699f0000 ffff8800699f02f8 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff81097d1c>] ? __call_rcu+0x110/0x118
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff81152a9f>] ? cap_file_free_security+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034d6780     0  9592  24433 0x00000000
[78137.427846]  ffff880001addbd0 0000000000000282 000000009b082000 
0000000000000002
[78137.427846]  0000000000000040 000000000000f9e0 ffff88004e16ffd8 
0000000000015780
[78137.427846]  0000000000015780 ffff8800699f2a60 ffff8800699f2d58 
00000004810414a4
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff81243032>] ? release_sock+0x13/0xa0
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff812fae21>] ? __mutex_lock_common+0x122/0x192
[78137.427846]  [<ffffffff812faf49>] ? mutex_lock+0x1a/0x31
[78137.427846]  [<ffffffff810f5ffc>] ? do_lookup+0x80/0x15d
[78137.427846]  [<ffffffff810f6a7c>] ? __link_path_walk+0x5a5/0x6f5
[78137.427846]  [<ffffffff810f6dfa>] ? path_walk+0x66/0xc9
[78137.427846]  [<ffffffff810f8264>] ? do_path_lookup+0x20/0x77
[78137.427846]  [<ffffffff810f9746>] ? user_path_at+0x48/0x79
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff810f1c22>] ? vfs_fstatat+0x2c/0x57
[78137.427846]  [<ffffffff810f1d15>] ? sys_newstat+0x11/0x30
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[78137.427846] apache2       D ffff88000345e780     0  9593  24433 0x00000000
[78137.427846]  ffff88005fff4db0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880028e37fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff8800699f54c0 ffff8800699f57b8 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff810ee6e1>] ? do_sync_read+0xce/0x113
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034d6780     0  9594  24433 0x00000000
[78137.427846]  ffff880086d84db0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800a2a19fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af3f90 ffff880001af4288 
0000000400000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846]  [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000345e780     0  9599  24433 0x00000000
[78137.427846]  ffff880001920710 0000000000000282 ffff880092bbbb38 
ffff880092bbbb34
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880092bbbfd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af1530 ffff880001af1828 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff810fcde3>] ? d_kill+0x58/0x61
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000347c780     0  9600  24433 0x00000000
[78137.427846]  ffff88009ad7e2e0 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800bbf79fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af2350 ffff880001af2648 
0000000100000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff810ee6e1>] ? do_sync_read+0xce/0x113
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000347c780     0  9601  24433 0x00000008
[78137.427846]  ffff880092e83880 0000000000000282 ffff88007664bb38 
ffff88007664bb34
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff88007664bfd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af0e20 ffff880001af1118 
0000000100000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8105363b>] ? _local_bh_enable_ip+0x22/0x8f
[78137.427846]  [<ffffffff8127b537>] ? inet_csk_accept+0x1bf/0x1e9
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034f4780     0  9604  24433 0x00000000
[78137.427846]  ffff8800018ae9f0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880072835fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af7100 ffff880001af73f8 
0000000500000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846]  [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000347c780     0  9606  24433 0x00000000
[78137.427846]  ffff880001878710 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800a6875fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af3170 ffff880001af3468 
0000000100000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff810f9750>] ? user_path_at+0x52/0x79
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000345e780     0  9608  24433 0x00000000
[78137.427846]  ffff88008392bf90 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff88001c965fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880001af1c40 ffff880001af1f38 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034d6780     0  9610  24433 0x00000000
[78137.427846]  ffff88005fcdc6a0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800066e3fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880092e87100 ffff880092e873f8 
0000000400000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff81289e23>] ? tcp_write_xmit+0x93e/0x96c
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8106c1df>] ? ktime_get_ts+0x68/0xb2
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D 0000000000000001     0  9611  24433 0x00000000
[78137.427846]  ffff8800bfbcd4c0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880057f57fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880092e84db0 ffff880092e850a8 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff810fcde3>] ? d_kill+0x58/0x61
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D 0000000000000000     0  9612  24433 0x00000000
[78137.427846]  ffff8800bfa58e20 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880087639fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880092e83f90 ffff880092e84288 
0000000100000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff81289e23>] ? tcp_write_xmit+0x93e/0x96c
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D 0000000000000000     0  9614  24433 0x00000000
[78137.427846]  ffff8800bfbcd4c0 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880024ffdfd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880092e82350 ffff880092e82648 
0000000400000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff810cc61f>] ? handle_mm_fault+0x27a/0x80f
[78137.427846]  [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034b8780     0  9616  24433 0x00000000
[78137.427846]  ffff880065e1f100 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880067231fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880092e80710 ffff880092e80a08 
0000000300000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff8100e5d4>] ? xen_smp_send_reschedule+0x0/0x7
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff81152ab5>] ? cap_dentry_open+0x0/0x3
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000347c780     0  9620  24433 0x00000000
[78137.427846]  ffff880002abc6a0 0000000000000286 0000000000000002 
ffffffff8103f604
[78137.427846]  0000000000000001 000000000000f9e0 ffff88008c655fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff88006c7a7810 ffff88006c7a7b08 
00000001810414a4
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8103f604>] ? find_busiest_queue+0xba/0xdb
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff812fae21>] ? __mutex_lock_common+0x122/0x192
[78137.427846]  [<ffffffff812faf49>] ? mutex_lock+0x1a/0x31
[78137.427846]  [<ffffffff810f5ffc>] ? do_lookup+0x80/0x15d
[78137.427846]  [<ffffffff810f6a7c>] ? __link_path_walk+0x5a5/0x6f5
[78137.427846]  [<ffffffff810f6dfa>] ? path_walk+0x66/0xc9
[78137.427846]  [<ffffffff810f8264>] ? do_path_lookup+0x20/0x77
[78137.427846]  [<ffffffff810f9746>] ? user_path_at+0x48/0x79
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff810f1c22>] ? vfs_fstatat+0x2c/0x57
[78137.427846]  [<ffffffff810f1d15>] ? sys_newstat+0x11/0x30
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[78137.427846] apache2       D ffff88000349a780     0  9625  24433 0x00000000
[78137.427846]  ffff880072080000 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880052299fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff880072082350 ffff880072082648 
0000000200000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff810cc61f>] ? handle_mm_fault+0x27a/0x80f
[78137.427846]  [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034f4780     0  9627  24433 0x00000000
[78137.427846]  ffff8800bd16e2e0 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff880066639fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff88000186a350 ffff88000186a648 
0000000500000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff81243032>] ? release_sock+0x13/0xa0
[78137.427846]  [<ffffffff8105363b>] ? _local_bh_enable_ip+0x22/0x8f
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff88000345e780     0  9629  24433 0x00000000
[78137.427846]  ffff880001923170 0000000000000282 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800a66e5fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff88009ad7aa60 ffff88009ad7ad58 
0000000000000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff810ee6e1>] ? do_sync_read+0xce/0x113
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff812fbf5a>] ? error_exit+0x2a/0x60
[78137.427846]  [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff8800034b8780     0  9630  24433 0x00000000
[78137.427846]  ffff880086d82a60 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff8800913e9fd8 
0000000000015780
[78137.427846]  0000000000015780 ffff88009ad7b170 ffff88009ad7b468 
0000000300000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff81289e23>] ? tcp_write_xmit+0x93e/0x96c
[78137.427846]  [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846]  [<ffffffff8106c1df>] ? ktime_get_ts+0x68/0xb2
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2       D ffff880003512780     0  9651  24433 0x00000000
[78137.427846]  ffff880002abc6a0 0000000000000286 0000000000000000 
000000010128ed1c
[78137.427846]  ffffffff8100dc35 000000000000f9e0 ffff88005dfeffd8 
0000000000015780
[78137.427846]  0000000000015780 ffff88009ad7a350 ffff88009ad7a648 
0000000600000200
[78137.427846] Call Trace:
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846]  [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846]  [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846]  [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846]  [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846]  [<ffffffff810bb6e5>] ? __do_page_cache_readahead+0x9b/0x1b4
[78137.427846]  [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846]  [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846]  [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846]  [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846]  [<ffffffff8100c241>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846]  [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846]  [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] Sched Debug Version: v0.09, 2.6.32-bpo.5-amd64 #1
[78137.427846] now at 78148825.414302 msecs
[78137.427846]   .jiffies                                 : 4314426651
[78137.427846]   .sysctl_sched_latency                    : 20.000000
[78137.427846]   .sysctl_sched_min_granularity            : 4.000000
[78137.427846]   .sysctl_sched_wakeup_granularity         : 4.000000
[78137.427846]   .sysctl_sched_child_runs_first           : 0.000000
[78137.427846]   .sysctl_sched_features                   : 15834235
[78137.427846]
[78137.427846] cpu#0, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 47902047
[78137.427846]   .nr_load_updates               : 15938294
[78137.427846]   .nr_uninterruptible            : 75
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78137422.159104
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[0]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 5680735.179660
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : 0.000000
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[0]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#1, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 8912456
[78137.427846]   .nr_load_updates               : 10191035
[78137.427846]   .nr_uninterruptible            : 30
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78148824.260676
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[1]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 3652776.005874
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -2027959.173786
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[1]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#2, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 6929803
[78137.427846]   .nr_load_updates               : 6509315
[78137.427846]   .nr_uninterruptible            : 20
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78148824.506917
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[2]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 3012547.394539
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -2668187.785121
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[2]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#3, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 4929509
[78137.427846]   .nr_load_updates               : 4016716
[78137.427846]   .nr_uninterruptible            : 13
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78149024.755812
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[3]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 2379542.626815
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -3301192.552845
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[3]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#4, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 3274386
[78137.427846]   .nr_load_updates               : 2348628
[78137.427846]   .nr_uninterruptible            : 33
[78137.427846]   .next_balance                  : 4314.426714
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78149025.003801
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[4]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 1864943.780970
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -3815791.398690
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[4]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#5, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 3594802
[78137.427846]   .nr_load_updates               : 1453667
[78137.427846]   .nr_uninterruptible            : 39
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78149025.261256
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[5]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 1282845.273539
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -4397889.906121
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[5]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#6, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 3841965
[78137.427846]   .nr_load_updates               : 1239272
[78137.427846]   .nr_uninterruptible            : 26
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78149025.516745
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[6]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 1124955.770063
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -4555779.409597
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[6]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#7, 2493.750 MHz
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_switches                   : 3495511
[78137.427846]   .nr_load_updates               : 999351
[78137.427846]   .nr_uninterruptible            : 15
[78137.427846]   .next_balance                  : 4314.426652
[78137.427846]   .curr->pid                     : 0
[78137.427846]   .clock                         : 78149221.757059
[78137.427846]   .cpu_load[0]                   : 0
[78137.427846]   .cpu_load[1]                   : 0
[78137.427846]   .cpu_load[2]                   : 0
[78137.427846]   .cpu_load[3]                   : 0
[78137.427846]   .cpu_load[4]                   : 0
[78137.427846]
[78137.427846] cfs_rq[7]:/
[78137.427846]   .exec_clock                    : 0.000000
[78137.427846]   .MIN_vruntime                  : 0.000001
[78137.427846]   .min_vruntime                  : 899194.224398
[78137.427846]   .max_vruntime                  : 0.000001
[78137.427846]   .spread                        : 0.000000
[78137.427846]   .spread0                       : -4781540.955262
[78137.427846]   .nr_running                    : 0
[78137.427846]   .load                          : 0
[78137.427846]   .nr_spread_over                : 0
[78137.427846]   .shares                        : 0
[78137.427846]
[78137.427846] rt_rq[7]:
[78137.427846]   .rt_nr_running                 : 0
[78137.427846]   .rt_throttled                  : 0
[78137.427846]   .rt_time                       : 0.000000
[78137.427846]   .rt_runtime                    : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846]             task   PID         tree-key  switches  prio     
exec-runtime         sum-exec        sum-sleep
[78137.427846] 
----------------------------------------------------------------------------------------------------------
[78137.427846]

And then I did:

[78270.883837] SysRq : Show backtrace of all active CPUs
[78270.883858] sending NMI to all CPUs:
[78270.883878] BUG: unable to handle kernel paging request at ffffffffff5fc310
[78270.883889] IP: [<ffffffff81027e23>] _flat_send_IPI_mask+0x4b/0x78
[78270.883905] PGD 1003067 PUD 1004067 PMD 159c067 PTE 0
[78270.883917] Thread overran stack, or stack corrupted
[78270.883929] Oops: 0002 [#1] SMP
[78270.883937] last sysfs file: 
/sys/devices/vbd-51713/block/xvda1/queue/scheduler
[78270.883947] CPU 0
[78270.883956] Modules linked in: ip6t_LOG ip6t_REJECT nf_conntrack_ipv6 
ip6table_filter ip6_tables xt_owner xt_recent ipt_REJECT ipt_LOG xt_state 
xt_tcpudp iptable_filter ip_tables x_tables nf_nat_ftp nf_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack snd_pcm snd_timer snd soundcore 
snd_page_alloc pcspkr xen_netfront evdev ext3 jbd mbcache dm_mirror 
dm_region_hash dm_log dm_snapshot dm_mod xen_blkfront thermal_sys
[78270.884063] Pid: 0, comm: swapper Not tainted 2.6.32-bpo.5-amd64 #1
[78270.884069] RIP: e030:[<ffffffff81027e23>]  [<ffffffff81027e23>] 
_flat_send_IPI_mask+0x4b/0x78
[78270.884079] RSP: e02b:ffff88000344cd88  EFLAGS: 00010086
[78270.884085] RAX: 00000000ff000000 RBX: 0000000000000800 RCX: 000000000000fe7d
[78270.884091] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 00000000000000ff
[78270.884097] RBP: 0000000000000002 R08: 000000000013f3fa R09: 000000000000000a
[78270.884103] R10: 0000000000000000 R11: 00000000000186a0 R12: ffffffff814d0040
[78270.884109] R13: 00000000000000ff R14: 0000000000000007 R15: 000000000000006c
[78270.884124] FS:  00007ffc218d0750(0000) GS:ffff880003449000(0000) 
knlGS:0000000000000000
[78270.884131] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[78270.884137] CR2: ffffffffff5fc310 CR3: 0000000092198000 CR4: 0000000000002620
[78270.884144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[78270.884150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[78270.884157] Process swapper (pid: 0, threadinfo ffffffff81428000, task 
ffffffff8145d1f0)
[78270.884164] Stack:
[78270.884168]  ffffffff8100e22f 0000000000000000 000000000000006c 
0000000000000001
[78270.884178] <0> ffff880002b09800 ffffffff8102506e 00000000000000f0 
ffffffff812fb8b4
[78270.884190] <0> ffffffff81493620 ffffffff8120c26d 0000000000000001 
ffff8800bd355200
[78270.884205] Call Trace:
[78270.884210]  <IRQ>
[78270.884217]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884225]  [<ffffffff8102506e>] ? arch_trigger_all_cpu_backtrace+0x3c/0x69
[78270.884235]  [<ffffffff812fb8b4>] ? _spin_lock_irqsave+0x15/0x34
[78270.884244]  [<ffffffff8120c26d>] ? __handle_sysrq+0xaf/0x14b
[78270.884251]  [<ffffffff8120cd3f>] ? hvc_poll+0x143/0x22c
[78270.884262]  [<ffffffff8120cf9d>] ? hvc_handle_interrupt+0xc/0x1f
[78270.884272]  [<ffffffff810945f0>] ? handle_IRQ_event+0x58/0x126
[78270.884281]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78270.884288]  [<ffffffff81095e34>] ? handle_edge_irq+0xc0/0x101
[78270.884296]  [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[78270.884305]  [<ffffffff811eeb8a>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[78270.884313]  [<ffffffff811eec4f>] ? xen_evtchn_do_upcall+0x2e/0x41
[78270.884320]  [<ffffffff81011cfe>] ? xen_do_hypervisor_callback+0x1e/0x30
[78270.884326]  <EOI>
[78270.884332]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884340]  [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884348]  [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884355]  [<ffffffff8100dcb3>] ? xen_safe_halt+0xc/0x15
[78270.884362]  [<ffffffff8100be63>] ? xen_idle+0x37/0x40
[78270.884372]  [<ffffffff8100feb1>] ? cpu_idle+0xa2/0xda
[78270.884383]  [<ffffffff814efcd1>] ? start_kernel+0x3dc/0x3e8
[78270.884389] Code: 48 8b 05 a1 a3 4a 00 83 fe 02 8b 58 34 75 0a ff 90 58 01 
00 00 eb 0e f3 90 8b 04 25 00 c3 5f ff f6 c4 10 75 f2 44 89 e8 c1 e0 18 <89> 04 
25 10 c3 5f ff 89 e8 09 d8 80 cf 04 83 fd 02 0f 44 c3 89
[78270.884482] RIP  [<ffffffff81027e23>] _flat_send_IPI_mask+0x4b/0x78
[78270.884491]  RSP <ffff88000344cd88>
[78270.884495] CR2: ffffffffff5fc310
[78270.884503] ---[ end trace 198fbc07a27eaf2a ]---
[78270.884509] Kernel panic - not syncing: Fatal exception in interrupt
[78270.884516] Pid: 0, comm: swapper Tainted: G      D    2.6.32-bpo.5-amd64 #1
[78270.884522] Call Trace:
[78270.884526]  <IRQ>  [<ffffffff812f9adc>] ? panic+0x86/0x143
[78270.884536]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884546]  [<ffffffff811b703c>] ? dummycon_dummy+0x0/0x3
[78270.884552]  [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78270.884560]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78270.884566]  [<ffffffff811b703c>] ? dummycon_dummy+0x0/0x3
[78270.884573]  [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78270.884580]  [<ffffffff812fc845>] ? oops_end+0xa7/0xb4
[78270.884588]  [<ffffffff810323e4>] ? no_context+0x1e9/0x1f8
[78270.884595]  [<ffffffff8100e242>] ? check_events+0x12/0x20
[78270.884602]  [<ffffffff81032599>] ? __bad_area_nosemaphore+0x1a6/0x1ca
[78270.884609]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884622]  [<ffffffff8104e8ef>] ? vprintk+0x321/0x364
[78270.884629]  [<ffffffff8100c205>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[78270.884641]  [<ffffffff81031d07>] ? spurious_fault+0x12c/0x1c4
[78270.884649]  [<ffffffff812fdc0f>] ? do_page_fault+0x69/0x2fc
[78270.884656]  [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78270.884663]  [<ffffffff81027e23>] ? _flat_send_IPI_mask+0x4b/0x78
[78270.884670]  [<ffffffff81027e0d>] ? _flat_send_IPI_mask+0x35/0x78
[78270.884677]  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884685]  [<ffffffff8102506e>] ? arch_trigger_all_cpu_backtrace+0x3c/0x69
[78270.884693]  [<ffffffff812fb8b4>] ? _spin_lock_irqsave+0x15/0x34
[78270.884706]  [<ffffffff8120c26d>] ? __handle_sysrq+0xaf/0x14b
[78270.884713]  [<ffffffff8120cd3f>] ? hvc_poll+0x143/0x22c
[78270.884720]  [<ffffffff8120cf9d>] ? hvc_handle_interrupt+0xc/0x1f
[78270.884731]  [<ffffffff810945f0>] ? handle_IRQ_event+0x58/0x126
[78270.884743]  [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78270.884750]  [<ffffffff81095e34>] ? handle_edge_irq+0xc0/0x101
[78270.884761]  [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[78270.884773]  [<ffffffff811eeb8a>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[78270.884780]  [<ffffffff811eec4f>] ? xen_evtchn_do_upcall+0x2e/0x41
[78270.884788]  [<ffffffff81011cfe>] ? xen_do_hypervisor_callback+0x1e/0x30
[78270.884793]  <EOI>  [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884804]  [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884811]  [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884818]  [<ffffffff8100dcb3>] ? xen_safe_halt+0xc/0x15
[78270.884825]  [<ffffffff8100be63>] ? xen_idle+0x37/0x40
[78270.884831]  [<ffffffff8100feb1>] ? cpu_idle+0xa2/0xda
[78270.887828]  [<ffffffff814efcd1>] ? start_kernel+0x3dc/0x3e8

Whoopsy daisy.

-- 
     2. That which causes joy or happiness.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.