Re: [Xen-devel] Crash during boot in Debian lenny default dom0 kernel (2.6.26-2-xen-686)

[George Dunlap <George.Dunlap@xxxxxxxxxxxxx>]

$ addr2line -e xen/xen-syms 0xffff82c4801535a9
/xensource/hg/open-source/xen-unstable.hg/xen/arch/x86/msi.c:588

Which in my code is:
    /* Mask interrupt here */
    writel(1, entry->mask_base + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET);

And tracing back, ->mask_base is computed in part with table_base
(which as Jan pointed out, seems borked).

Hmm... Jan said according to the stack:
.table_base = f5861e4a00000001

Is it possible that there's actually a bug in the compat code, and
that table_base actually *was* set to (uint32_t)1?  If a reasonable
number for table_base is "1", giving it 64 bits in the structure would
seem a bit like overkill...

 -George

On Thu, Feb 25, 2010 at 6:50 AM, Jiang, Yunhong <yunhong.jiang@xxxxxxxxx> wrote:
> Agree that Xen should not crash on such situation.
>
> Since it is enabling MSI, I try to check the code in __pci_enable_msi(), and 
> didn't find any suspicious code. As 0x ffff82c3ffeb614d is at ioremap range, 
> I suspect it is about msix fixmap, but still didn't find any hint.
> The page table walk is a bit strange to me. The L1 entry is suspicious, seems 
> it clobered.
> Can you share the code around the fault IP address?
>
> --jyh
>
>
> (XEN) Pagetable walk from ffff82c3ffeb614d:
> (XEN)  L4[0x105] = 00000000bf4e2027 5555555555555555
> (XEN)  L3[0x10f] = 00000000bf698063 5555555555555555
> (XEN)  L2[0x1ff] = 00000000bf697063 5555555555555555
> (XEN)  L1[0x0b6] = f5861e4a00100173 ffffffffffffffff
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 15:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=000b]
> (XEN) Faulting linear address: ffff82c3ffeb614d
> (XEN) ****************************************
>
>
>
>>-----Original Message-----
>>From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>>[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of George Dunlap
>>Sent: Thursday, February 25, 2010 7:58 AM
>>To: Pasi Kärkkäinen
>>Cc: Sander Eikelenboom; xen-devel@xxxxxxxxxxxxxxxxxxx
>>Subject: Re: [Xen-devel] Crash during boot in Debian lenny default dom0 kernel
>>(2.6.26-2-xen-686)
>>
>>pci=nomsi in the guest command line?
>>
>>I realize dom0 is a privileged guest, but it still seems like we
>>should try not to crash Xen as a result on guest input. :-)
>>
>>Thanks for the work-around,
>> -George
>>
>>On Wed, Feb 24, 2010 at 8:20 PM, Pasi Kärkkäinen <pasik@xxxxxx> wrote:
>>> On Wed, Feb 24, 2010 at 08:08:10PM +0100, Sander Eikelenboom wrote:
>>>>
>>>> Wasn't pci=nomsi required for those kernels ?
>>>>
>>>
>>> Yeah, pci=nomsi has been the workaround for this bug..
>>> Many people have seen (and reported) this bug in the lenny kernel.
>>>
>>> Would be good to hunt down the actual reason..
>>>
>>> -- Pasi
>>>
>>>>
>>>> Wednesday, February 24, 2010, 7:47:09 PM, you wrote:
>>>>
>>>> > I recenty tried to boot my Debian lenny distro on a newer Intel
>>>> > Nehalem box using Debian Lenny's default xen kernel
>>>> > (2.6.26-2-xen-686), and it crashed during boot.  Full log attached,
>>>> > but the key info is here:
>>>>
>>>> > I tried doing a "binary search" to see where this was introduced, but
>>>> > it fails the same way all the way back to c/s 20000.
>>>>
>>>> > The same kernel/hypervisor combination boots fine on a different box I 
>>>> > have.
>>>>
>>>> >  -George
>>>>
>>>> > (XEN) ----[ Xen-4.0.0-rc4  x86_64  debug=n  Tainted:    C ]----
>>>> > (XEN) CPU:    15
>>>> > (XEN) RIP:    e008:[<ffff82c4801535a9>] pci_enable_msi+0x4c9/0x580
>>>> > (XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
>>>> > (XEN) rax: ffff82c3ffeb6141   rbx: 0000000000000000   rcx:
>>ffff830828b0e690
>>>> > (XEN) rdx: ffff830828b0e718   rsi: ffff8300bf6975b0   rdi: 
>>>> > ffff830828b0e6e0
>>>> > (XEN) rbp: ffff83082cd7fec8   rsp:
>>ffff83082cd7fd88   r8:  0000000000000005
>>>> > (XEN) r9:  ffff830828b0e700   r10: 00000000000000a8   r11:
>>0000000000000040
>>>> > (XEN) r12: ffff830828b0e710   r13: ffff830828b0e670   r14: 
>>>> > ffff83082cd7fe38
>>>> > (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4:
>>00000000000026f0
>>>> > (XEN) cr3: 000000043fdcf000   cr2: ffff82c3ffeb614d
>>>> > (XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0000   cs: e008
>>>> > (XEN) Xen stack trace from rsp=ffff83082cd7fd88:
>>>> > (XEN)    0000000000000246 ffff830828b0e6e0 0000000080000000
>>0000000100000000
>>>> > (XEN)    3408000000000000 0000000200000000 000f5861e4a00100
>>0000000000000141
>>>> > (XEN)    ffff83082cd71ec8 ffff83082cd7fec8 0000000000000136
>>ffff83082cd60000
>>>> > (XEN)    0000000000000038 00000000ffffffed 0000000000000000
>>ffff82c480154832
>>>> > (XEN)    00000000000004d8 0000000000000038 00000000000000e0
>>ffff83083fd81c80
>>>> > (XEN)    ffff830828b0e670 00000000f5861e0c 0000000000000000
>>ffff83082cd60000
>>>> > (XEN)    ffff83082cd400f0 00000000f5861e0c 0000000000000136
>>0000000000000038
>>>> > (XEN)    ffff83082cd7fec8 ffff82c4801ede5f 7265646e776f206f
>>ffff82c48024c080
>>>> > (XEN)    ffff83082cd60180 ffff83082cd7fe98 0000000000007ff0 
>>>> > ffffffffffffffff
>>>> > (XEN)    0000000800000000 0000000100000000 ffff8308f5861e4a
>>ffff82c4802eaa80
>>>> > (XEN)    0000000800000000 0000000000000038 f5861e4a00000001
>>000000000000000f
>>>> > (XEN)    ffffffffffffffff ffff8300bf2e8000 0000000000000035
>>0000000000000000
>>>> > (XEN)    0000000000000000 0000000000000000 0000000000000000
>>ffff82c4801ef863
>>>> > (XEN)    0000000000000000 0000000000000000 0000000000000000
>>0000000000000000
>>>> > (XEN)    0000000000000035 000000000000000d 0000000000000000
>>0000000000000000
>>>> > (XEN)    0000000000000000 0000000000000000 0000000000000021
>>00000000f5861e0c
>>>> > (XEN)    00000000c033741f 00000000ffffffff 0000000000000000
>>0000010000000000
>>>> > (XEN)    00000000c0101427 0000000000000061 0000000000000286
>>00000000f5861e08
>>>> > (XEN)    0000000000000069 0000000000000000 0000000000000000
>>0000000000000000
>>>> > (XEN)    0000000000000000 000000000000000f ffff8300bf2e8000
>>>> > (XEN) Xen call trace:
>>>> > (XEN)    [<ffff82c4801535a9>] pci_enable_msi+0x4c9/0x580
>>>> > (XEN)    [<ffff82c480154832>] map_domain_pirq+0x242/0x2f0
>>>> > (XEN)    [<ffff82c4801ede5f>] compat_physdev_op+0xc8f/0x1010
>>>> > (XEN)    [<ffff82c4801ef863>] compat_hypercall+0x83/0x90
>>>> > (XEN)
>>>> > (XEN) Pagetable walk from ffff82c3ffeb614d:
>>>> > (XEN)  L4[0x105] = 00000000bf4e2027 5555555555555555
>>>> > (XEN)  L3[0x10f] = 00000000bf698063 5555555555555555
>>>> > (XEN)  L2[0x1ff] = 00000000bf697063 5555555555555555
>>>> > (XEN)  L1[0x0b6] = f5861e4a00100173 ffffffffffffffff
>>>> > (XEN)
>>>> > (XEN) ****************************************
>>>> > (XEN) Panic on CPU 15:
>>>> > (XEN) FATAL PAGE FAULT
>>>> > (XEN) [error_code=000b]
>>>> > (XEN) Faulting linear address: ffff82c3ffeb614d
>>>> > (XEN) ****************************************
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>  Sander                            mailto:linux@xxxxxxxxxxxxxx
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>> http://lists.xensource.com/xen-devel
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel
>>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@xxxxxxxxxxxxxxxxxxx
>>http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel