Re: [Xen-devel] [xen-unstable test] 11946: regressions - FAIL

[AP <apxeng@xxxxxxxxx>]

On Fri, May 4, 2012 at 8:11 PM, Andrew Cooper <andrew.cooper3@xxxxxxxxxx> wrote:
>
> On 04/05/12 20:48, AP wrote:
> > On Tue, Mar 27, 2012 at 3:36 AM, Ian Campbell <Ian.Campbell@xxxxxxxxxx>
> > wrote:
> >> On Tue, 2012-02-14 at 10:44 +0000, Ian Campbell wrote:
> >>> On Mon, 2012-02-13 at 20:16 +0000, xen.org wrote:
> >>>> flight 11946 xen-unstable real [real]
> >>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/11946/
> >>>>
> >>>> Regressions :-(
> >>>>
> >>>> Tests which did not succeed and are blocking,
> >>>> including tests which could not be run:
> >>>>  test-amd64-i386-xl-credit2    7 debian-install            fail REGR.
> >>>> vs. 11944
> >>> Host crash:
> >>>
> >>> http://www.chiark.greenend.org.uk/~xensrcts/logs/11946/test-amd64-i386-xl-credit2/serial-woodlouse.log
> >>>
> >>> This is the debug Andrew Cooper added recently to track down the IRQ
> >>> assertion we've been seeing, sadly it looks like the debug code tries
> >>> to
> >>> call xfree from interrupt context and therefore doesn't produce full
> >>> output :-(
> >> Are we still seeing the issue this debugging was intended to address?
> >> We
> >> don't seem to be seeing the host crashes any more. Should the debug
> >> code
> >> be patched up as in the following patch, otherwise when we do see it it
> >> doesn't end up printing any useful info.
> >>
> >> Someone recently reported bugs.debian.org/665433 to Debian, is this the
> >> same underlying issue? That report is with Xen 4.0 FWIW.
> > I saw the issue (xen-unstable 25256:9dda0efd8ce1) that the debugging
> > code added. Can the fix to the debugging code be checked in until the
> > original issue has been fixed?
> >
> > Thanks,
> > AP
> >
> > (XEN) *** IRQ BUG found ***
> > (XEN) CPU0 -Testing vector 236 from bitmap
> >
> > 41,47,49,57,64,72,80,88,96,100,104,120,136,152,160-161,168,171,192,200-201,208
> > (XEN) Guest interrupt information:
> > (XEN)    IRQ:   0 affinity:01 vec:f0 type=IO-APIC-edge
> > status=00000000 mapped, unbound
> > (XEN) Assertion '!in_irq()' failed at xmalloc_tlsf.c:607
> > (XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Tainted:    C ]----
> > (XEN) CPU:    0
> > (XEN) RIP:    e008:[<ffff82c48012cefb>] xfree+0x33/0x118
> > (XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
> > (XEN) rax: 0000000000000000   rbx: ffff830214ac0080   rcx:
> > 0000000000000000
> > (XEN) rdx: ffff82c4802d8880   rsi: 0000000000000083   rdi:
> > 0000000000000000
> > (XEN) rbp: ffff82c4802b7c78   rsp: ffff82c4802b7c58   r8:
> >  0000000000000004
> > (XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
> > 0000000000000010
> > (XEN) r12: ffff830214ac0c80   r13: 000000000000000c   r14:
> > ffff830214ac0ca8
> > (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4:
> > 00000000000426f0
> > (XEN) cr3: 0000000168971000   cr2: 0000000001095e00
> > (XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
> > (XEN) Xen stack trace from rsp=ffff82c4802b7c58:
> > (XEN)    ffff830214ac0080 ffff830214ac0c80 000000000000000c
> > ffff830214ac0ca8
> > (XEN)    ffff82c4802b7ce8 ffff82c4801664d4 ffff82c4802e214a
> > ffff82c400000020
> > (XEN)    ffff82c4802b7cf8 0000000000000083 ffff830214ac00a8
> > 0000000000000000
> > (XEN)    00000000000000ec 00000000000000ec ffff830214ac0c80
> > 000000000000000c
> > (XEN)    ffff830214ac0ca8 ffff82c480302760 ffff82c4802b7d58
> > ffff82c480168000
> > (XEN)    ffff82c4802b7f18 ffff82c4802b7f18 000000ec00000000
> > ffff82c4802b7f18
> > (XEN)    0000000000000000 0000000000000000 ffff82c480302324
> > 0000000000000020
> > (XEN)    ffff82c4802b7dd8 0000000000000003 0000000000000000
> > 0000000000000000
> > (XEN)    ffff82c4802b7dc8 ffff82c4801683d3 ffff8300da991000
> > ffff8300da996000
> > (XEN)    0000000000000000 ffffffff802b7d90 ffff82c480159160
> > ffff82c4802b7e20
> > (XEN)    ffff82c48015d7db ffff82c4802b7f18 ffff8300da991000
> > 0000000000000003
> > (XEN)    0000000000000000 0000000000000000 00007d3b7fd48207
> > ffff82c480160426
> > (XEN)    0000000000000000 0000000000000000 0000000000000003
> > ffff8300da991000
> > (XEN)    ffff82c4802b7ef8 ffff82c4802b7f18 0000000000000282
> > ffff82c4802319a0
> > (XEN)    00000000deadbeef 0000000000000000 ffff83021c0b8081
> > 0000000000000000
> > (XEN)    0000000000000048 ffff8801d7227ec0 ffff8300da991000
> > 0000002000000000
> > (XEN)    ffff82c4801865c1 000000000000e008 0000000000000202
> > ffff82c4802b7e88
> > (XEN)    000000000000e010 0000000000000003 ffff82c4802b7ef8
> > ffff82c4802230d8
> > (XEN)    ffff82c4802b7f18 0000000000000000 0000000000000246
> > ffffffff810013aa
> > (XEN)    0000000000000000 ffffffff810013aa 000000000000e030
> > 0000000000000246
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82c48012cefb>] xfree+0x33/0x118
> > (XEN)    [<ffff82c4801664d4>] dump_irqs+0x2a4/0x2e8
> > (XEN)    [<ffff82c480168000>] irq_move_cleanup_interrupt+0x29f/0x2db
> > (XEN)    [<ffff82c4801683d3>] do_IRQ+0x9e/0x5a4
> > (XEN)    [<ffff82c480160426>] common_interrupt+0x26/0x30
> > (XEN)    [<ffff82c4801865c1>] async_exception_cleanup+0x1/0x35a
> > (XEN)    [<ffff82c480228438>] syscall_enter+0xc8/0x122
> > (XEN)
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Assertion '!in_irq()' failed at xmalloc_tlsf.c:607
> > (XEN) ****************************************
> > (XEN)
> > (XEN) Reboot in five seconds...
> The attached patch should prevent this panic, allowing for all the debug
> information to be printed to the console.

Thanks, that fixed it. Here is what I see now:

(XEN) *** IRQ BUG found ***
(XEN) CPU0 -Testing vector 236 from bitmap 37,41,49,51,64,72,80,88,96,104,120,136,145,152,158,160,168,175,182,192,200,211
(XEN) Guest interrupt information:
(XEN)    IRQ:   0 affinity:01 vec:f0 type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   1 affinity:01 vec:d3 type=IO-APIC-edge    status=00000030 in-flight=0 domain-list=0:  1(-S--),
(XEN)    IRQ:   2 affinity:ff vec:e2 type=XT-PIC          status=00000000 mapped, unbound
(XEN)    IRQ:   3 affinity:01 vec:40 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   4 affinity:01 vec:48 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   5 affinity:01 vec:50 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   6 affinity:01 vec:58 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   7 affinity:01 vec:60 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   8 affinity:08 vec:29 type=IO-APIC-edge    status=00000030 in-flight=0 domain-list=0:  8(-S--),
(XEN)    IRQ:   9 affinity:02 vec:25 type=IO-APIC-level   status=00000030 in-flight=0 domain-list=0:  9(-S--),
(XEN)    IRQ:  10 affinity:01 vec:78 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  11 affinity:01 vec:88 type=IO-APIC-edge    status=00000002 mapped, unbound
[ 5129.737147] [drm:i915_hangcheck_ring_idle] *ERROR* Hangcheck timer elapsed... blt ring idle [waiting on 1800652, at 1800652], missed IRQ?

Let me know if you need any more info.
Thanks,
AP

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel