[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen 3.2.1-rc5: FATAL PAGE FAULT



Keir Fraser wrote:
That's going to be rather hard to answer without finding a more reliable
repro of the bug, and then adding some hypervisor tracing.

Here are two more Xen traces with this problem. These always appear to occur after we're forced to destroy a domain. The first trace is a DoubleDump<tm> and has something new in the second dump...

http://www.theshore.net/~caker/xen/build-1.11/

I still don't have a method to reproduce, but since we're hitting this with some frequency, would it be worth it to stick in some extra debugging now?


====== First trace ======

----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
CPU:    1
RIP:    e008:[<ffff828c8013dee4>] put_page_type+0x17/0x107
RFLAGS: 0000000000210286   CONTEXT: hypervisor
rax: 00001da2f4162bf0   rbx: ffffa026f4162bf0   rcx: 0000000080000000
rdx: 000000bdac808de6   rsi: 0000000000402fe3   rdi: ffffa026f4162bf0
rbp: ffff8300cf13fbf8   rsp: ffff8300cf13fbd8   r8:  0000000000000000
r9:  00000000deadbeef   r10: ffff828c801c5bf0   r11: 0000000000000000
r12: 0000000000000000   r13: ffffa026f4162bf0   r14: 0000000000402fe3
r15: ffff82840a077b78   cr0: 000000008005003b   cr4: 00000000000026b0
cr3: 000000062ffdd000   cr2: ffffa026f4162c08
ds: 007b   es: 007b   fs: 0000   gs: 0033   ss: 0000   cs: e008
Xen stack trace from rsp=ffff8300cf13fbd8:
   0000000000000002 ffffa026f4162bf0 0000000000000000 ffff8300cee48100
   ffff8300cf13fc18 ffff828c8013b3bb 0000000000200202 ffff830402fe3000
   ffff8300cf13fc58 ffff828c8013bfcd 00000000cee48100 ffff8300cee48100
   ffff82840a077b78 000000004c000001 ffff8300cee48100 ffff8300cee48118
   ffff8300cf13fc88 ffff828c8013de4a 0000000044000001 000000004c000000
   ffff82840a077b78 ffff8300cee48100 ffff8300cf13fcb8 ffff828c8013df63
   00007cff30ec0337 ffff82840a077b78 0000000000000003 00000000004011a4
   ffff8300cf13fcd8 ffff828c8013b409 ffff8300cf13fd68 ffff8304011a4018
   ffff8300cf13fd08 ffff828c8013c06d ffff8300cee48100 ffff82840a02c1a0
   0000000068000001 ffff8300cee490f8 ffff8300cf13fd38 ffff828c8013de5a
   0000000060000001 0000000068000000 ffff82840a02c1a0 ffff8300cee490f8
   ffff8300cf13fd68 ffff828c8013df63 ffff82840a02c1a0 ffff82840a02c1a0
   ffff82840a02c1a0 ffff8300cee48100 ffff8300cf13fdb8 ffff828c80131680
   0000000088000000 0000000080000000 ffff8300cf13ff28 ffff8300cee48100
   ffff8300cee48100 00000000b4dfc508 0000000000000000 0000000000000000
   ffff8300cf13fdd8 ffff828c80131a94 ffff8300cee48100 0000000000000000
   ffff8300cf13fe08 ffff828c80105638 ffff82840f448b58 ffff8300cf13fe28
   00000000b4dfc508 fffffffffffffff3 ffff8300cf13ff08 ffff828c8010479f
   00000000000000fb ffff8300cee3a130 ffff8300cf13fe68 ffff828c8011c746
   0000000000200282 ffff8300ceefe118 0000000500000002 083010acb7ab000a
Xen call trace:
   [<ffff828c8013dee4>] put_page_type+0x17/0x107
   [<ffff828c8013b3bb>] put_page_from_l2e+0x3f/0x4e
   [<ffff828c8013bfcd>] free_l2_table+0xa6/0xce
   [<ffff828c8013de4a>] free_page_type+0x1c4/0x247
   [<ffff828c8013df63>] put_page_type+0x96/0x107
   [<ffff828c8013b409>] put_page_from_l3e+0x3f/0x4e
   [<ffff828c8013c06d>] free_l3_table+0x78/0xc4
   [<ffff828c8013de5a>] free_page_type+0x1d4/0x247
   [<ffff828c8013df63>] put_page_type+0x96/0x107
   [<ffff828c80131680>] relinquish_memory+0xce/0x262
   [<ffff828c80131a94>] domain_relinquish_resources+0xd1/0x1b0
   [<ffff828c80105638>] domain_kill+0x77/0x164
   [<ffff828c8010479f>] do_domctl+0x4dd/0xc1e
   [<ffff828c801bdd50>] compat_tracing_off+0xb/0x64

Pagetable walk from ffffa026f4162c08:
 L4[0x140] = 0000000000000000 ffffffffffffffff

****************************************
Panic on CPU 1:
FATAL PAGE FAULT
[error_code=0000]
Faulting linear address: ffffa026f4162c08
****************************************

Reboot in five seconds...

...3 seconds later, this occurred...

Assertion '__cpus_subset(&(cpumask), &(cpu_online_map), 32)' failed at smp.c:84
----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
CPU:    0
RIP:    e008:[<ffff828c80145c68>] send_IPI_mask_flat+0x29/0x9c
RFLAGS: 0000000000010002   CONTEXT: hypervisor
rax: 00000000fffffffe   rbx: ffff8300cee3c100   rcx: 0000000000000003
rdx: 0000000000000040   rsi: 00000000000000fc   rdi: 0000000000000004
rbp: ffff828c80237be8   rsp: ffff828c80237bd0   r8:  ffff828c8024c780
r9:  0000000000000002   r10: 00000000deadbeef   r11: 0000000000000000
r12: 0000000000000004   r13: 00000000000000fc   r14: 0000000000000010
r15: 00001485db7a5091   cr0: 000000008005003b   cr4: 00000000000026b0
cr3: 00000003ff15a000   cr2: 00000000e3015078
ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0000   cs: e008
Xen stack trace from rsp=ffff828c80237bd0:
   ffff8300cee3c100 0000000000000086 0000000000000000 ffff828c80237c08
   ffff828c8014601a ffff8300cee30f00 0000000000000004 ffff828c80237c38
   ffff828c80114da0 0000000000000004 ffff828c80137fe0 0000000000000004
   ffff828c8025951c ffff828c80237c68 ffff828c80119b18 ffff828c80237c98
   ffff828c80137ac2 ffff8300cee3c100 ffff8300cfdd4100 ffff828c80237c98
   ffff828c80107409 00000000c0621300 ffff8300cfdd4100 ffff8300cee30f00
   0000000000000000 ffff828c80237ca8 ffff828c801075c9 ffff828c80237cd8
   ffff828c80137fe0 ffff828c80259500 ffff828c8025951c 0000000000000098
   ffff828c80237d38 ffff828c80237d28 ffff828c80137ac2 0000000000000082
   0000000000000000 ffff828c80237d18 0000000000000009 00000000ffffffff
   ffff828c801ebb60 ffff828c8020e100 00001485db7a5091 00007d737fdc82a7
   ffff828c801336e6 00001485db7a5091 ffff828c8020e100 ffff828c801ebb60
   00000000ffffffff ffff828c80237de8 0000000000000009 0000000000000000
   00000000deadbeef 0000000000000000 0000000000000000 000000007d9b040e
   000000007d8a4358 000000000000290c 00000000001e8480 00000000000003e8
   0000009800000000 ffff828c8012ac48 000000000000e008 0000000000000216
   ffff828c80237de8 0000000000000000 00001485db7a5091 ffff828c80237e08
   ffff828c80146257 ffff828c80237f28 ffff828c8020e534 ffff828c80237e28
   ffff828c80145b9a ffff828c80237f28 ffff828c8020e534 ffff828c80237e38
   ffff828c80146312 00007d737fdc8197 ffff828c801347a0 00001485db7a5091
Xen call trace:
   [<ffff828c80145c68>] send_IPI_mask_flat+0x29/0x9c
   [<ffff828c8014601a>] smp_send_event_check_mask+0x3e/0x40
   [<ffff828c80114da0>] csched_vcpu_wake+0x242/0x259
   [<ffff828c80119b18>] vcpu_wake+0x12d/0x248
   [<ffff828c80107409>] evtchn_set_pending+0xe5/0x15c
   [<ffff828c801075c9>] send_guest_pirq+0x61/0x63
   [<ffff828c80137fe0>] __do_IRQ_guest+0x19c/0x1b2
   [<ffff828c80137ac2>] do_IRQ+0x5a/0x1a7
   [<ffff828c801336e6>] common_interrupt+0x26/0x30
   [<ffff828c8012ac48>] __udelay+0x30/0x48
   [<ffff828c80146257>] smp_send_stop+0x39/0x67
   [<ffff828c80145b9a>] machine_restart+0x4f/0xc5
   [<ffff828c80146312>] smp_call_function_interrupt+0x79/0xa7
   [<ffff828c801347a0>] call_function_interrupt+0x30/0x40
   [<ffff828c8012c73b>] default_idle+0x2f/0x34
   [<ffff828c8012c7ff>] idle_loop+0x70/0x77


****************************************
Panic on CPU 0:
Assertion '__cpus_subset(&(cpumask), &(cpu_online_map), 32)' failed at smp.c:84
****************************************

Reboot in five seconds...


====== Second trace ======

----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
CPU:    0
RIP:    e008:[<ffff828c8013dee4>] put_page_type+0x17/0x107
RFLAGS: 0000000000210286   CONTEXT: hypervisor
rax: 00000a51169fd050   rbx: ffff8cd5169fd050   rcx: 0000000080000000
rdx: 0000004206f73202   rsi: 00000000004041e1   rdi: ffff8cd5169fd050
rbp: ffff828c80237bf8   rsp: ffff828c80237bd8   r8:  0000000000000000
r9:  00000000deadbeef   r10: ffff828c801c5bf0   r11: 0000000000000000
r12: 0000000000000000   r13: ffff8cd5169fd050   r14: 00000000004041e1
r15: ffff82840a0a4b28   cr0: 000000008005003b   cr4: 00000000000026b0
cr3: 000000062ffd9000   cr2: ffff8cd5169fd068
ds: 007b   es: 007b   fs: 0000   gs: 0033   ss: 0000   cs: e008
Xen stack trace from rsp=ffff828c80237bd8:
   ffff828409df5d01 ffff8cd5169fd050 0000000000000000 ffff8300ceea0100
   ffff828c80237c18 ffff828c8013b3bb 0000000400000004 ffff8304041e1000
   ffff828c80237c58 ffff828c8013bfcd 00000003f2f24027 ffff8300ceea0100
   ffff82840a0a4b28 0000000048000001 ffff8300ceea0100 ffff8300ceea0118
   ffff828c80237c88 ffff828c8013de4a 0000000040000001 0000000048000000
   ffff82840a0a4b28 ffff8300ceea0100 ffff828c80237cb8 ffff828c8013df63
   0000000000000000 ffff82840a0a4b28 0000000000000000 0000000000402dd4
   ffff828c80237cd8 ffff828c8013b409 ffff8300ceea0100 ffff830402dd4000
   ffff828c80237d08 ffff828c8013c06d ffff8300ceea0100 ffff82840a072920
   0000000068000001 ffff8300ceea10f8 ffff828c80237d38 ffff828c8013de5a
   0000000060000001 0000000068000000 ffff82840a072920 ffff8300ceea10f8
   ffff828c80237d68 ffff828c8013df63 ffff82840a072920 ffff82840a072920
   ffff82840a072920 ffff8300ceea0100 ffff828c80237db8 ffff828c80131680
   0000000088000000 0000000080000000 ffff828c80237f28 ffff8300ceea0100
   ffff8300ceea0100 00000000b2cf9868 0000000000000000 0000000000000000
   ffff828c80237dd8 ffff828c80131a94 ffff8300ceea0100 0000000000000000
   ffff828c80237e08 ffff828c80105638 ffff828c80237e18 ffff828c80114da0
   00000000b2cf9868 fffffffffffffff3 ffff828c80237f08 ffff828c8010479f
   ffff828c80237e48 ffff8300cee36130 0000000000000000 000078cdfb20f27f
   0000000000200282 0000000000000000 0000000500000002 081d66ecb7af0010
Xen call trace:
   [<ffff828c8013dee4>] put_page_type+0x17/0x107
   [<ffff828c8013b3bb>] put_page_from_l2e+0x3f/0x4e
   [<ffff828c8013bfcd>] free_l2_table+0xa6/0xce
   [<ffff828c8013de4a>] free_page_type+0x1c4/0x247
   [<ffff828c8013df63>] put_page_type+0x96/0x107
   [<ffff828c8013b409>] put_page_from_l3e+0x3f/0x4e
   [<ffff828c8013c06d>] free_l3_table+0x78/0xc4
   [<ffff828c8013de5a>] free_page_type+0x1d4/0x247
   [<ffff828c8013df63>] put_page_type+0x96/0x107
   [<ffff828c80131680>] relinquish_memory+0xce/0x262
   [<ffff828c80131a94>] domain_relinquish_resources+0xd1/0x1b0
   [<ffff828c80105638>] domain_kill+0x77/0x164
   [<ffff828c8010479f>] do_domctl+0x4dd/0xc1e
   [<ffff828c801bdd50>] compat_tracing_off+0xb/0x64

Pagetable walk from ffff8cd5169fd068:
 L4[0x119] = 0000000000000000 ffffffffffffffff

****************************************
Panic on CPU 0:
FATAL PAGE FAULT
[error_code=0000]
Faulting linear address: ffff8cd5169fd068
****************************************

Reboot in five seconds...

-Chris

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.