Re: [Xen-devel] [linux-linus bisection] complete test-amd64-amd64-qemuu-nested-intel

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Tue, Mar 17, 2020 at 06:22:55AM +0000, Tian, Kevin wrote:
> > From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
> > Sent: Saturday, March 14, 2020 12:03 AM
> > 
> > On 12/03/2020 23:56, osstest service owner wrote:
> > > branch xen-unstable
> > > xenbranch xen-unstable
> > > job test-amd64-amd64-qemuu-nested-intel
> > > testid debian-hvm-install/l1/l2
> > >
> > > Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-
> > 2.6.git
> > > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> > > Tree: ovmf git://xenbits.xen.org/osstest/ovmf.git
> > > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> > > Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> > > Tree: seabios git://xenbits.xen.org/osstest/seabios.git
> > > Tree: xen git://xenbits.xen.org/xen.git
> > >
> > > *** Found and reproduced problem changeset ***
> > >
> > >   Bug is in tree:  xen git://xenbits.xen.org/xen.git
> > >   Bug introduced:  f96e1469ad06b61796c60193daaeb9f8a96d7458
> > >   Bug not present: 0729830cc425a8ff27a3137e87b93768ae3c853c
> > >   Last fail repro: 
> > > http://logs.test-lab.xenproject.org/osstest/logs/148496/
> > >
> > >
> > >   commit f96e1469ad06b61796c60193daaeb9f8a96d7458
> > >   Author: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> > >   Date:   Wed Feb 5 13:49:09 2020 +0100
> > >
> > >       x86/vvmx: fix virtual interrupt injection when Ack on exit control 
> > > is used
> > >
> > >       When doing a virtual vmexit (ie: a vmexit handled by the L1 VMM)
> > >       interrupts shouldn't be injected using the virtual interrupt 
> > > delivery
> > >       mechanism unless the Ack on exit vmexit control bit isn't set in the
> > >       nested vmcs.
> > >
> > >       Gate the call to nvmx_update_apicv helper on whether the nested vmcs
> > >       has the Ack on exit bit set in the vmexit control field.
> > >
> > >       Note that this fixes the usage of x2APIC by the L1 VMM, at least 
> > > when
> > >       the L1 VMM is Xen.
> > >
> > >       Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> > >       Reviewed-by: Kevin Tian <kevin.tian@xxxxxxxxx>
> > 
> > Looks like there are further problems here.  I'm afraid I haven't
> > investigated further, but this also might be the source of the
> > intermittent problems in staging.
> > 
> 
> any error message or stack dump for this failure?

Sorry, forgot to reply yesterday. I'm already looking into it, the
crash seems to be from L1 getting wrong interrupt injection:

(XEN) Assertion '!sp || (peoi[sp - 1].vector < vector)' failed at irq.c:1862
(XEN) ----[ Xen-4.14-unstable  x86_64  debug=y   Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82d080289dfe>] do_IRQ+0x4e3/0x6f8
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor (d1v0)
(XEN) rax: 000000000000006e   rbx: 000000000000006e   rcx: ffff8300b93212a0
(XEN) rdx: 0000000000000001   rsi: 0000000000000001   rdi: ffff8300b9321280
(XEN) rbp: ffff8300bf5efdb8   rsp: ffff8300bf5efd48   r8:  0000000000000001
(XEN) r9:  ffff8300bf5b6068   r10: 0000166c3548742a   r11: 000000550f69eeb0
(XEN) r12: ffff8300bf291d40   r13: 000000000000006e   r14: ffff8300bf5c2000
(XEN) r15: ffff8300b93214a0   cr0: 0000000080050033   cr4: 00000000003526e0
(XEN) cr3: 00000000bf28c000   cr2: ffff8a4b06f3a000
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d080289dfe> (do_IRQ+0x4e3/0x6f8):
(XEN)  39 c3 0f 87 6c ff ff ff <0f> 0b 0f 0b 0f 0b b8 00 00 00 00 eb 49 0f ae e8
(XEN) Xen stack trace from rsp=ffff8300bf5efd48:
(XEN)    ffff82d08038e851 0000003038d61000 ffff82d000000000 ffff8300bf5c2024
(XEN)    0000000000000000 000000208038e845 ffff82d08038e851 ffff82d08038e845
(XEN)    ffff82d08038e851 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffff8300bf5effff 0000000000000000 00007cff40a10217 ffff82d08038e8ba
(XEN)    ffff8300bf290000 0000000000000000 0000000000000001 ffff8300b92bc000
(XEN)    ffff8300bf5efee8 ffff8300bf5efef8 0000000000000019 000000000001aa55
(XEN)    000000550caca034 ffff8300bf291d50 ffff82d0805c07e8 0000000000000000
(XEN)    0000003038d61000 0000000000000000 ffff8300bf290000 0000006e00000000
(XEN)    ffff82d080335c04 000000000000e008 0000000000000286 ffff8300bf5efe78
(XEN)    0000000000000000 ffff82d080335b29 ffff82d08033ccd1 ffff82d08033ccc5
(XEN)    0000000000000000 ffff82d08033ccc5 ffff82d08033ccd1 ffff82d08033ccc5
(XEN)    ffff82d08033ccd1 ffff82d08033ccc5 ffff82d08033ccd1 ffff8300b92bc000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    00007cff40a100e7 ffff82d08033cd1a 000000000000008e ffff8a4b2d18cedc
(XEN)    ffffffffa2482f86 00000000000003c8 ffffffffa2b1a120 ffff8a4b2d18cebe
(XEN)    ffff8a4b2d19cfa0 ffff8a4b000b8000 ffff8a4b2d19c000 0000000000000720
(XEN)    000000000000002a 0000000000000190 00000000000003c9 ffffffffa2482f80
(XEN)    ffff8a4b2d18cc00 0000006e0000beef ffffffffa1f9dad7 000000bf0000beef
(XEN)    0000000000000246 ffffffffa2803ec8 000000000000beef e486997731cdbeef
(XEN)    085780c33cb2beef 5d42c695b7d9beef 4fa29714ad64beef 0000e01000000001
(XEN) Xen call trace:
(XEN)    [<ffff82d080289dfe>] R do_IRQ+0x4e3/0x6f8
(XEN)    [<ffff82d08038e851>] S common_interrupt+0xa1/0x120
(XEN)    [<ffff82d08038e8ba>] F common_interrupt+0x10a/0x120
(XEN)    [<ffff82d080335c04>] F vmx_vmexit_handler+0x32b/0x1e60
(XEN)    [<ffff82d08033cd1a>] F vmx_asm_vmexit_handler+0xfa/0x270
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) Assertion '!sp || (peoi[sp - 1].vector < vector)' failed at irq.c:1862
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

This is however not reproducible on all boxes, as it needs a special
set of VMX features available AFAIK in order to trigger it. I've
managed to get such a box and I'm debugging it.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel