Xen project Mailing List

Re: [Xen-devel] 4.11.0 RC1 panic

From: Manuel Bouyer <bouyer@xxxxxxxxxxxxxxx>

Date: Sun, 10 Jun 2018 12:57:28 +0200

Delivery-date: Sun, 10 Jun 2018 10:57:48 +0000

List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Sun, Jun 10, 2018 at 03:54:45AM -0600, Jan Beulich wrote: > [...] > > So I've been trying to look into this some more, and I've noticed an oddity in > the raw stack dump you had provided with the first report. Unfortunately you > didn't include that part for either of the above (the first one as being on a > different path would be of particular interest). > Additionally, to be able to check > whether the (type_info) values on the stack really point at some anomaly, I'd > need the xen-syms (or xen.efi) from that same build of Xen. That'll allow me > to determine whether the values are simply leftovers from prior function > invocations. Otherwise, if they're live values, it'd be suspicious for > free_page_type() to be called on a page the type refcount of which is still 2. > > As I assume that you don't have recorded/stored the additional bits I'm asking > for, I do realize that this means that unfortunately you'll have to obtain > the data > another time. I'm sorry for that. > Actually I have them: I have complete logs of the serial console, and I still have the built directory. This one is from a newer build (4.11.0rc4). I've put the binary files at ftp://asim.lip6.fr/outgoing/bouyer/xen-debug/ (XEN) Xen version 4.11-rcnb0 (bouyer@) (gcc (nb2 20150115) 4.8.5) debug=y Tue M ay 15 17:21:40 MEST 2018 [...] (XEN) Assertion 'oc > 0' failed at mm.c:681 (XEN) ----[ Xen-4.11-rcnb0 x86_64 debug=y Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20 (XEN) RFLAGS: 0000000000010246 CONTEXT: hypervisor (d0v0) (XEN) rax: ffffffffffff0000 rbx: 4400000000000001 rcx: 00000000001a3200 (XEN) rdx: 0400000000000000 rsi: 000000000000002e rdi: ffff82e003465040 (XEN) rbp: ffff82e003464000 rsp: ffff8301bf127bb0 r8: 0000000000000000 (XEN) r9: 0000000000000200 r10: 4000000000000000 r11: ffff82e003465040 (XEN) r12: ffff82e003465040 r13: 0000000000000000 r14: 10ffffffffffffff (XEN) r15: 1000000000000000 cr0: 000000008005003b cr4: 0000000000002660 (XEN) cr3: 00000001b9096000 cr2: 00007f7ff60ce790 (XEN) fsb: 00007f7ff7ff36c0 gsb: ffffffff80cc8500 gss: 0000000000000000 (XEN) ds: 003f es: 003f fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around <ffff82d080284bd2> (mm.c#dec_linear_entries+0x12/0x20): (XEN) c1 47 1e 66 85 c0 7f 02 <0f> 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00 41 54 (XEN) Xen stack trace from rsp=ffff8301bf127bb0: (XEN) ffff82d08028922e 000000000023a00d ffff8301bf127fff ffff82d08023a00d (XEN) ffff82e003464000 ffff82e003465040 ffff8301b98b6000 0000000000000001 (XEN) ffff820040030000 0200000000000000 ffff82d0802898af 00000000000001fc (XEN) ffff82e003465040 ffff82d080288c59 00000000001a3282 0000001500000000 (XEN) ffff8301bf127fff 4400000000000001 ffff82e003465040 0000000000000000 (XEN) 00ffffffffffffff 10ffffffffffffff 1000000000000000 ffff82d0802891f7 (XEN) 000000010122471f ffff8301bf127fff ffff8301bf127d10 ffff82e003465040 (XEN) ffff8301bf127d10 ffff8301b98b6028 ffff8301b98b6000 ffff8301b98b6020 (XEN) ffff82e003465050 ffff82d0802898ef ffff82d080272adb 0000000000000068 (XEN) e400000000000000 ffff8301bf127fff 8000000000000000 ffff8301b98b6000 (XEN) 0000000000000000 ffff8301b98b6018 deadbeefdeadf00d 0000000000000001 (XEN) 00007f7ff7b1b004 ffff82d080276ae3 ffff8301b98b6000 00007f7ff7b1b004 (XEN) ffff82d0802068bd ffff8301b98b6000 00007f7ff7b1b004 0000000000000000 (XEN) ffff82d0802039e3 ffff82d080457640 ffff8301bf0e3fa0 0000000000000002 (XEN) 00000000000000f0 0000000000000000 0000000000000000 0000000000000000 (XEN) 000000000000000f 0000000000000000 0000001000000002 00007f7ff7c00016 (XEN) 00007f7ff7ffe400 0000000000000016 00007f7fffffd510 00007f7ff74c97d1 (XEN) 00007f7ff7c02d7d 0000000000000246 0000000000000000 0000000000000000 (XEN) 0000000000000000 00007f7ff7b10800 0000000000000016 0000000000000016 (XEN) 0000000000000000 00007f7ff7b10800 0000000000000202 00007f7ff7ffa800 (XEN) Xen call trace: (XEN) [<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20 (XEN) [<ffff82d08028922e>] mm.c#_put_page_type+0x13e/0x350 (XEN) [<ffff82d08023a00d>] _spin_lock+0xd/0x50 (XEN) [<ffff82d0802898af>] mm.c#put_page_from_l2e+0xdf/0x110 (XEN) [<ffff82d080288c59>] free_page_type+0x2f9/0x790 (XEN) [<ffff82d0802891f7>] mm.c#_put_page_type+0x107/0x350 (XEN) [<ffff82d0802898ef>] put_page_type_preemptible+0xf/0x10 (XEN) [<ffff82d080272adb>] domain.c#relinquish_memory+0xab/0x460 (XEN) [<ffff82d080276ae3>] domain_relinquish_resources+0x203/0x290 (XEN) [<ffff82d0802068bd>] domain_kill+0xbd/0x150 (XEN) [<ffff82d0802039e3>] do_domctl+0x7d3/0x1a90 (XEN) [<ffff82d080203210>] do_domctl+0/0x1a90 (XEN) [<ffff82d080367b95>] pv_hypercall+0x1f5/0x430 (XEN) [<ffff82d08036e422>] lstar_enter+0xa2/0x120 (XEN) [<ffff82d08036e42e>] lstar_enter+0xae/0x120 (XEN) [<ffff82d08036e422>] lstar_enter+0xa2/0x120 (XEN) [<ffff82d08036e42e>] lstar_enter+0xae/0x120 (XEN) [<ffff82d08036e422>] lstar_enter+0xa2/0x120 (XEN) [<ffff82d08036e42e>] lstar_enter+0xae/0x120 (XEN) [<ffff82d08036e48c>] lstar_enter+0x10c/0x120 The second one is from the same build: (XEN) Xen version 4.11-rcnb0 (bouyer@) (gcc (nb2 20150115) 4.8.5) debug=y Tue M ay 15 17:21:40 MEST 2018 [...] (XEN) Assertion 'oc > 0' failed at mm.c:681 (XEN) ----[ Xen-4.11-rcnb0 x86_64 debug=y Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20 (XEN) RFLAGS: 0000000000010246 CONTEXT: hypervisor (d2v0) (XEN) rax: ffffffffffff0000 rbx: 4400000000000001 rcx: 00000000001a7d29 (XEN) rdx: 0400000000000000 rsi: 0000000000000011 rdi: ffff82e0034f5540 (XEN) rbp: ffff82e0034fa520 rsp: ffff8301bf13fc08 r8: 0000000000000000 (XEN) r9: 0000000000000200 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: ffff82e0034f5540 r13: 0000000000000000 r14: 10ffffffffffffff (XEN) r15: 1000000000000000 cr0: 000000008005003b cr4: 00000000000026e4 (XEN) cr3: 00000001b98b6000 cr2: 00000000bb418010 (XEN) fsb: 00000000c0636dc0 gsb: 0000000000000000 gss: 0000000000000000 (XEN) ds: 0011 es: 0011 fs: 0031 gs: 0011 ss: 0000 cs: e008 (XEN) Xen code around <ffff82d080284bd2> (mm.c#dec_linear_entries+0x12/0x20): (XEN) c1 47 1e 66 85 c0 7f 02 <0f> 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00 41 54 (XEN) Xen stack trace from rsp=ffff8301bf13fc08: (XEN) ffff82d08028922e 0000000000800063 ffff8301bf13ffff 4c00000000000002 (XEN) ffff82e0034fa520 ffff82e0034f5540 ffff8301b98f5000 0000000000000001 (XEN) ffff820040018000 0200000000000000 ffff82d0802898af 00000000000001fd (XEN) ffff82e0034f5540 ffff82d080288c59 00000000001a7aaa 0000000000000000 (XEN) ffff8301bf13ffff 4400000000000001 ffff82e0034f5540 0000000000000000 (XEN) 00ffffffffffffff 10ffffffffffffff 1000000000000000 ffff82d0802891f7 (XEN) 0000000101000206 ffff8301bf13ffff 4400000000000002 00000000001a7aaa (XEN) ffff82e0034f5540 0000000000000000 ffff8301b98f5000 ffff820080000000 (XEN) 0000000000000000 ffff82d0802898ef ffff82d080290b6d ffff8300bff40000 (XEN) 00000001802a8302 ffff8301b98f5000 0000000000000000 ffff8301b98f5000 (XEN) 0000000000000007 ffff8300bff40000 00007ff000000000 0000000000000000 (XEN) ffff8301b98f5000 ffff82e00350bf80 ffff82d0804b0058 ffff82d0804b0060 (XEN) 00000000001a85fc 00000000001a85fc 0000000100000004 00000000001a7aaa (XEN) 00000000001aaaa7 ffff820080000018 0000000000000001 00000000cdb0db7c (XEN) ffff82d080387b90 0000000000000001 ffff8301bf13ffff ffff82d080295630 (XEN) ffff8301bf13fe14 00000001bf13ffff ffff820080000000 0000000000000000 (XEN) 00007ff000000000 000000048036bbe8 cdb0dbb0001a7aaa ffff8301bf13fef8 (XEN) ffff8300bff40000 00000000000001a0 00000000deadf00d 0000000000000004 (XEN) 00000000deadf00d ffff82d080367d4a ffff82d000007ff0 ffff82d000000000 (XEN) ffff82d000000001 ffff82d0cdb0db70 ffff82d08036bbf4 ffff82d08036bbe8 (XEN) Xen call trace: (XEN) Xen call trace: (XEN) [<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20 (XEN) [<ffff82d08028922e>] mm.c#_put_page_type+0x13e/0x350 (XEN) [<ffff82d0802898af>] mm.c#put_page_from_l2e+0xdf/0x110 (XEN) [<ffff82d080288c59>] free_page_type+0x2f9/0x790 (XEN) [<ffff82d0802891f7>] mm.c#_put_page_type+0x107/0x350 (XEN) [<ffff82d0802898ef>] put_page_type_preemptible+0xf/0x10 (XEN) [<ffff82d080290b6d>] do_mmuext_op+0x73d/0x1810 (XEN) [<ffff82d080295630>] compat_mmuext_op+0x430/0x450 (XEN) [<ffff82d080367d4a>] pv_hypercall+0x3aa/0x430 (XEN) [<ffff82d08036bbf4>] entry_int82+0x74/0xc0 (XEN) [<ffff82d08036bbe8>] entry_int82+0x68/0xc0 (XEN) [<ffff82d08036bbf4>] entry_int82+0x74/0xc0 (XEN) [<ffff82d08036bbe8>] entry_int82+0x68/0xc0 (XEN) [<ffff82d08036bbf4>] entry_int82+0x74/0xc0 (XEN) [<ffff82d08036bbe8>] entry_int82+0x68/0xc0 (XEN) [<ffff82d08036bbf4>] entry_int82+0x74/0xc0 (XEN) [<ffff82d08036957e>] do_entry_int82+0x1e/0x20 (XEN) [<ffff82d08036bc31>] entry_int82+0xb1/0xc0 > > Two other questions on the internals of NetBSD page table management: Are all > updates to a given (set of) page table(s) fully serialized, e.g. via a > respective > spin lock? In the general case they're done using atomic operations (pmap_pte_cas()). For xenPV, the update is protected by a global lock, so page table updates are serialized (globally). > Could you additionally point me at that code, or give an outline of > how the (un)pinning of L2 tables in the 32-bit case works? When a new set of page tables is needed (this is pmap_create()), a pdp is requested from a cache. If the cache is empty, pages are allocated in pmap_pdp_ctor(), which is going to also pin the L2 pages. When the page table is not needed any more (this is pmap_destroy()), the pdp is returned to the cache. L2 pages remains pinned, with pointers to the kernel L1 pages. If memory needs to be reclaimed from the cache, or is an explicit call to pool_cache_destruct_object() is done, the L2 pages are unpinned, but they are not explicitely zeroed out before (can this be a problem ?). The code for this is in http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/x86/x86/pmap.c Some helper functions are in http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/xen/x86/xen_pmap.c Some #defines and inline are in http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/xen/include/xenpmap.h http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/i386/include/pmap.h -- Manuel Bouyer <bouyer@xxxxxxxxxxxxxxx> NetBSD: 26 ans d'experience feront toujours la difference -- _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.