[Xen-users] Kernel paging request crash

[Paul Arthur <flowerysong00@xxxxxxxxx>]

Using pv_ops kernel (2.6.30-rc3-tip-13092-g52535f4) under 64-bit Gentoo
in Dom0, vanilla 2.6.30 under 32-bit LFS in DomU.

The guest is running iozone to generate a load, and eventually locks up.

The host shows a message like the following in 'xm dmesg':

(XEN) mm.c:2307:d43 Bad type (saw 2800000000000001 != exp e000000000000000) for
mfn d2b32 (pfn 71130)
(XEN) mm.c:804:d43 Error getting mfn d2b32 (pfn 71130) from L1 entry
80000000d2b32063 for dom43
(XEN) mm.c:4195:d43 ptwr_emulate: could not get_page_from_l1e()

(XEN) mm.c:2306:d47 Bad type (saw 2800000000000001 != exp e000000000000000) for
mfn d96fe (pfn 7f501)
(XEN) mm.c:804:d47 Error getting mfn d96fe (pfn 7f501) from L1 entry
80000000d96fe063 for dom47
(XEN) mm.c:4195:d47 ptwr_emulate: could not get_page_from_l1e()


The guest first reports:

client1 kernel: BUG: unable to handle kernel paging request at c070fd30
client1 kernel: IP: [<c01040ac>] xen_set_pte+0x1c/0x30
client1 kernel: *pdpt = 0000000249e69027 
client1 kernel: Oops: 0003 [#1] SMP 
client1 kernel: last sysfs file: 
client1 kernel: Modules linked in: xt_state xt_tcpudp iptable_raw 
iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 
iptable_filter ip_tables x_tables 8139too mii
client1 kernel: 
client1 kernel: Pid: 218, comm: kswapd0 Not tainted (2.6.30 #1) 
client1 kernel: EIP: 0061:[<c01040ac>] EFLAGS: 00010292 CPU: 1
client1 kernel: EIP is at xen_set_pte+0x1c/0x30
client1 kernel: EAX: 00000000 EBX: d2b32063 ECX: 80000000 EDX: 22b13000
client1 kernel: ESI: 80000000 EDI: c070fd30 EBP: f57ff000 ESP: e2543d68
client1 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
client1 kernel: Process kswapd0 (pid: 218, ti=e2542000 task=e24b4b30 
task.ti=e2542000)
client1 kernel: Stack:
client1 kernel: 80000000 c070fd30 000000a0 c0127024 80000000 00000000 d2b32063 
80000000
client1 kernel: 00000001 00000001 c06f0060 000000a0 00000014 00000163 80000000 
00000007
client1 kernel: b7ef1000 71130067 e2543e1c c0103c08 00000163 80000000 00000000 
c017ef19
client1 kernel: Call Trace:
client1 kernel: [<c0127024>] ? kmap_atomic_prot+0x114/0x150
client1 kernel: [<c0103c08>] ? xen_kmap_atomic_pte+0x28/0x40
client1 kernel: [<c017ef19>] ? page_check_address+0xb9/0x1a0
client1 kernel: [<c011f305>] ? arch_flush_lazy_mmu_mode+0x5/0x40
client1 kernel: [<c017f0df>] ? page_referenced_one+0x5f/0xf0
client1 kernel: [<c017ffbc>] ? page_referenced+0xec/0x120
client1 kernel: [<c016ce1f>] ? shrink_active_list+0x11f/0x390
client1 kernel: [<c01035d1>] ? xen_mc_flush+0xe1/0x190
client1 kernel: [<c0105337>] ? xen_force_evtchn_callback+0x17/0x30
client1 kernel: [<c0105b44>] ? check_events+0x8/0xc
client1 kernel: [<c016ded9>] ? shrink_zone+0x269/0x330
client1 kernel: [<c016ea15>] ? kswapd+0x555/0x5a0
client1 kernel: [<c016c540>] ? isolate_pages_global+0x0/0x210
client1 kernel: [<c0148e10>] ? autoremove_wake_function+0x0/0x50
client1 kernel: [<c016e4c0>] ? kswapd+0x0/0x5a0
client1 kernel: [<c0148b09>] ? kthread+0x49/0x80
client1 kernel: [<c0148ac0>] ? kthread+0x0/0x80
client1 kernel: [<c0108897>] ? kernel_thread_helper+0x7/0x10
client1 kernel: Code: eb 0d 90 90 90 90 90 90 90 90 90 90 90 90 90 83 ec 0c 89 
74 24 04 89 ce 89 7c 24 08 89 c7 89 1c 24 89 d3 e8 67 ae 01 00 89 77 04 <89> 1f 
8b 74 24 04 8b 1c 24 8b 7c 24 08 83 c4 0c c3 8d 76 00 56 
client1 kernel: EIP: [<c01040ac>] xen_set_pte+0x1c/0x30 SS:ESP 0069:e2543d68
client1 kernel: CR2: 00000000c070fd30


Then a series of messages like:

client1 kernel: ------------[ cut here ]------------
client1 kernel: kernel BUG at arch/x86/mm/highmem_32.c:45!
client1 kernel: invalid opcode: 0000 [#2] SMP 
client1 kernel: last sysfs file: 
client1 kernel: Modules linked in: xt_state xt_tcpudp iptable_raw 
iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 
iptable_filter ip_tables x_tables 8139too mii
client1 kernel: 
client1 kernel: Pid: 7973, comm: ping Tainted: G      D    (2.6.30 #1) 
client1 kernel: EIP: 0061:[<c012705b>] EFLAGS: 00010286 CPU: 1
client1 kernel: EIP is at kmap_atomic_prot+0x14b/0x150
client1 kernel: EAX: 00000000 EBX: 80000000 ECX: c1febea0 EDX: 80000000
client1 kernel: ESI: 7f5f5067 EDI: c1000000 EBP: f57ff000 ESP: e18e5c9c
client1 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
client1 kernel: Process ping (pid: 7973, ti=e18e4000 task=e1dc8c30 
task.ti=e18e4000)
client1 kernel: Stack:
client1 kernel: 00000002 ce46d067 00000000 00000000 00000000 e1823e84 c01776d6 
000000a0
client1 kernel: 00000014 00000161 80000000 00000007 7f5f5067 c1000000 00000007 
c0103c08
client1 kernel: 00000161 80000000 00000000 c017788e e25ce7d4 c030a3ae 0000001a 
c030a5e5
client1 kernel: Call Trace:
client1 kernel: [<c01776d6>] ? __pte_alloc+0x46/0x110
client1 kernel: [<c0103c08>] ? xen_kmap_atomic_pte+0x28/0x40
client1 kernel: [<c017788e>] ? handle_mm_fault+0xee/0xa10
client1 kernel: [<c030a3ae>] ? prio_tree_remove+0x6e/0xc0
client1 kernel: [<c030a5e5>] ? prio_tree_insert+0x1e5/0x250
client1 kernel: [<c017c6bd>] ? vma_link+0x7d/0xa0
client1 kernel: [<c01213a5>] ? do_page_fault+0xf5/0x2c0
client1 kernel: [<c01212b0>] ? do_page_fault+0x0/0x2c0
client1 kernel: [<c04e2ee5>] ? error_code+0x6d/0x74
client1 kernel: [<c01b00d8>] ? bio_kmalloc+0x8/0x50
client1 kernel: [<c01212b0>] ? do_page_fault+0x0/0x2c0
client1 kernel: [<c030f64f>] ? clear_user+0x2f/0x50
client1 kernel: [<c01bf5cc>] ? padzero+0x1c/0x30
client1 kernel: [<c01c1b66>] ? load_elf_binary+0x15c6/0x1820
client1 kernel: [<c0103786>] ? __raw_callee_save_xen_pte_val+0x6/0x8
client1 kernel: [<c0175b9b>] ? vm_normal_page+0x2b/0xb0
client1 kernel: [<c011f305>] ? arch_flush_lazy_mmu_mode+0x5/0x40
client1 kernel: [<c0126ef4>] ? kunmap_atomic+0x54/0x70
client1 kernel: [<c01782eb>] ? __get_user_pages+0x13b/0x440
client1 kernel: [<c0148e10>] ? autoremove_wake_function+0x0/0x50
client1 kernel: [<c01926ca>] ? get_arg_page+0x4a/0xb0
client1 kernel: [<c01c05a0>] ? load_elf_binary+0x0/0x1820
client1 kernel: [<c01929aa>] ? search_binary_handler+0x9a/0x240
client1 kernel: [<c0193f17>] ? do_execve+0x227/0x2a0
client1 kernel: [<c01066d1>] ? sys_execve+0x31/0x70
client1 kernel: [<c0107e58>] ? sysenter_do_call+0x12/0x2c
client1 kernel: Code: 2c 5b 5e 83 c0 45 c1 e0 0c 5f 29 c5 89 e8 5d c3 a1 88 e0 
69 c0 8b 74 24 24 8b 7c 24 28 21 c6 a1 8c e0 69 c0 21 c7 e9 69 ff ff ff <0f> 0b 
eb fe 90 53 83 ec 08 8b 1d 1c b7 71 c0 8b 0d 18 b7 71 c0 
client1 kernel: EIP: [<c012705b>] kmap_atomic_prot+0x14b/0x150 SS:ESP 
0069:e18e5c9c
client1 kernel: ---[ end trace b52d0922024c229a ]---
client1 kernel: Fixing recursive fault but reboot is needed!
client1 kernel: BUG: scheduling while atomic: ping/7973/0x00000002
client1 kernel: Modules linked in: xt_state xt_tcpudp iptable_raw 
iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 
iptable_filter ip_tables x_tables 8139too mii
client1 kernel: Pid: 7973, comm: ping Tainted: G      D    2.6.30 #1
client1 kernel: Call Trace:
client1 kernel: [<c04e0f19>] ? __schedule+0x2c9/0x6f0
client1 kernel: [<c0105b3b>] ? xen_restore_fl_direct_end+0x0/0x1
client1 kernel: [<c0136484>] ? vprintk+0x204/0x320
client1 kernel: [<c04e1345>] ? schedule+0x5/0x20
client1 kernel: [<c0138b99>] ? do_exit+0x649/0x660
client1 kernel: [<c0105b3b>] ? xen_restore_fl_direct_end+0x0/0x1
client1 kernel: [<c01092d0>] ? do_invalid_op+0x0/0xa0
client1 kernel: [<c01365bb>] ? printk+0x1b/0x20
client1 kernel: [<c01092d0>] ? do_invalid_op+0x0/0xa0
client1 kernel: [<c010b51f>] ? oops_end+0x9f/0xa0
client1 kernel: [<c0109351>] ? do_invalid_op+0x81/0xa0
client1 kernel: [<c012705b>] ? kmap_atomic_prot+0x14b/0x150
client1 kernel: [<c036df9a>] ? vt_console_print+0x20a/0x2d0
client1 kernel: [<c0105337>] ? xen_force_evtchn_callback+0x17/0x30
client1 kernel: [<c0105b44>] ? check_events+0x8/0xc
client1 kernel: [<c0105b3b>] ? xen_restore_fl_direct_end+0x0/0x1
client1 kernel: [<c04e2ee5>] ? error_code+0x6d/0x74
client1 kernel: [<c013007b>] ? task_utime+0x9b/0xb0
client1 kernel: [<c01092d0>] ? do_invalid_op+0x0/0xa0
client1 kernel: [<c012705b>] ? kmap_atomic_prot+0x14b/0x150
client1 kernel: [<c0105b44>] ? check_events+0x8/0xc
client1 kernel: [<c0105b3b>] ? xen_restore_fl_direct_end+0x0/0x1
client1 kernel: [<c01035d1>] ? xen_mc_flush+0xe1/0x190
client1 kernel: [<c0103c08>] ? xen_kmap_atomic_pte+0x28/0x40
client1 kernel: [<c0176b06>] ? unmap_vmas+0x2d6/0x840
client1 kernel: [<c0105b44>] ? check_events+0x8/0xc
client1 kernel: [<c0105b3b>] ? xen_restore_fl_direct_end+0x0/0x1
client1 kernel: [<c017b947>] ? exit_mmap+0x87/0x140
client1 kernel: [<c0133783>] ? mmput+0x33/0xa0
client1 kernel: [<c0137210>] ? exit_mm+0xd0/0x100
client1 kernel: [<c04e2c22>] ? _spin_lock_irq+0x12/0x20
client1 kernel: [<c0138638>] ? do_exit+0xe8/0x660
client1 kernel: [<c01092d0>] ? do_invalid_op+0x0/0xa0
client1 kernel: [<c01092d0>] ? do_invalid_op+0x0/0xa0
client1 kernel: [<c010b51f>] ? oops_end+0x9f/0xa0
client1 kernel: [<c0109351>] ? do_invalid_op+0x81/0xa0
client1 kernel: [<c012705b>] ? kmap_atomic_prot+0x14b/0x150
client1 kernel: [<c01052ed>] ? xen_flush_tlb_single+0x6d/0x80
client1 kernel: [<c011f305>] ? arch_flush_lazy_mmu_mode+0x5/0x40
client1 kernel: [<c0126ef4>] ? kunmap_atomic+0x54/0x70
client1 kernel: [<c016815d>] ? get_page_from_freelist+0x2cd/0x490
client1 kernel: [<c0105337>] ? xen_force_evtchn_callback+0x17/0x30
client1 kernel: [<c0105b44>] ? check_events+0x8/0xc
client1 kernel: [<c04e2ee5>] ? error_code+0x6d/0x74
client1 kernel: [<c01092d0>] ? do_invalid_op+0x0/0xa0
client1 kernel: [<c012705b>] ? kmap_atomic_prot+0x14b/0x150
client1 kernel: [<c01776d6>] ? __pte_alloc+0x46/0x110
client1 kernel: [<c0103c08>] ? xen_kmap_atomic_pte+0x28/0x40
client1 kernel: [<c017788e>] ? handle_mm_fault+0xee/0xa10
client1 kernel: [<c030a3ae>] ? prio_tree_remove+0x6e/0xc0
client1 kernel: [<c030a5e5>] ? prio_tree_insert+0x1e5/0x250
client1 kernel: [<c017c6bd>] ? vma_link+0x7d/0xa0
client1 kernel: [<c01213a5>] ? do_page_fault+0xf5/0x2c0
client1 kernel: [<c01212b0>] ? do_page_fault+0x0/0x2c0
client1 kernel: [<c04e2ee5>] ? error_code+0x6d/0x74
client1 kernel: [<c01b00d8>] ? bio_kmalloc+0x8/0x50
client1 kernel: [<c01212b0>] ? do_page_fault+0x0/0x2c0
client1 kernel: [<c030f64f>] ? clear_user+0x2f/0x50
client1 kernel: [<c01bf5cc>] ? padzero+0x1c/0x30
client1 kernel: [<c01c1b66>] ? load_elf_binary+0x15c6/0x1820
client1 kernel: [<c0103786>] ? __raw_callee_save_xen_pte_val+0x6/0x8
client1 kernel: [<c0175b9b>] ? vm_normal_page+0x2b/0xb0
client1 kernel: [<c011f305>] ? arch_flush_lazy_mmu_mode+0x5/0x40
client1 kernel: [<c0126ef4>] ? kunmap_atomic+0x54/0x70
client1 kernel: [<c01782eb>] ? __get_user_pages+0x13b/0x440
client1 kernel: [<c0148e10>] ? autoremove_wake_function+0x0/0x50
client1 kernel: [<c01926ca>] ? get_arg_page+0x4a/0xb0
client1 kernel: [<c01c05a0>] ? load_elf_binary+0x0/0x1820
client1 kernel: [<c01929aa>] ? search_binary_handler+0x9a/0x240
client1 kernel: [<c0193f17>] ? do_execve+0x227/0x2a0
client1 kernel: [<c01066d1>] ? sys_execve+0x31/0x70
client1 kernel: [<c0107e58>] ? sysenter_do_call+0x12/0x2c


client1 kernel: BUG: soft lockup - CPU#3 stuck for 61s! [sh:7974]
client1 kernel: Modules linked in: xt_state xt_tcpudp iptable_raw 
iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 
iptable_filter ip_tables x_tables 8139too mii
client1 kernel: 
client1 kernel: Pid: 7974, comm: sh Tainted: G      D    (2.6.30 #1) 
client1 kernel: EIP: 0061:[<c01013a7>] EFLAGS: 00000246 CPU: 3
client1 kernel: EIP is at _stext+0x3a7/0x1010
client1 kernel: EAX: 00000000 EBX: 00000003 ECX: e1dd7ccc EDX: 00000016
client1 kernel: ESI: e1db19a8 EDI: 00000000 EBP: e1db19aa ESP: e1dd7cc8
client1 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
client1 kernel: CR0: 8005003b CR2: b7dbcbcc CR3: 218e2000 CR4: 00002660
client1 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
client1 kernel: DR6: ffff0ff0 DR7: 00000400
client1 kernel: Call Trace:
client1 kernel: [<c0356a7d>] ? xen_poll_irq+0x3d/0x50
client1 kernel: [<c01063fb>] ? xen_spin_lock_slow+0x7b/0xf0
client1 kernel: [<c0106539>] ? xen_spin_lock+0x49/0x90
client1 kernel: [<c04e2b35>] ? _spin_lock+0x5/0x10
client1 kernel: [<c017fd0f>] ? anon_vma_unlink+0x1f/0x80
client1 kernel: [<c0179a1d>] ? free_pgtables+0x5d/0xc0
client1 kernel: [<c017b978>] ? exit_mmap+0xb8/0x140
client1 kernel: [<c0133783>] ? mmput+0x33/0xa0
client1 kernel: [<c01937aa>] ? flush_old_exec+0x39a/0x6a0
client1 kernel: [<c018efc1>] ? vfs_read+0x101/0x130
client1 kernel: [<c018e680>] ? do_sync_read+0x0/0x110
client1 kernel: [<c0192e3b>] ? kernel_read+0x3b/0x60
client1 kernel: [<c01c0807>] ? load_elf_binary+0x267/0x1820
client1 kernel: [<c01683db>] ? __alloc_pages_internal+0x9b/0x430
client1 kernel: [<c0177f08>] ? handle_mm_fault+0x768/0xa10
client1 kernel: [<c0103786>] ? __raw_callee_save_xen_pte_val+0x6/0x8
client1 kernel: [<c0175b9b>] ? vm_normal_page+0x2b/0xb0
client1 kernel: [<c011f305>] ? arch_flush_lazy_mmu_mode+0x5/0x40
client1 kernel: [<c0126ef4>] ? kunmap_atomic+0x54/0x70
client1 kernel: [<c01767ea>] ? follow_page+0x2ca/0x310
client1 kernel: [<c01782eb>] ? __get_user_pages+0x13b/0x440
client1 kernel: [<c0148e10>] ? autoremove_wake_function+0x0/0x50
client1 kernel: [<c01926ca>] ? get_arg_page+0x4a/0xb0
client1 kernel: [<c01c05a0>] ? load_elf_binary+0x0/0x1820
client1 kernel: [<c01929aa>] ? search_binary_handler+0x9a/0x240
client1 kernel: [<c0193f17>] ? do_execve+0x227/0x2a0
client1 kernel: [<c01066d1>] ? sys_execve+0x31/0x70
client1 kernel: [<c0107e58>] ? sysenter_do_call+0x12/0x2c

-- 
Well, some take delight in the carriages a-rolling,
And some take delight in the hurling and the bowling,
But I take delight in the juice of the barley,
And courting pretty fair maids in the morning bright and early.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users