WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] Re: xen dependant on pcpu 0 ?

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: [Xen-devel] Re: xen dependant on pcpu 0 ?
From: Sander Eikelenboom <linux@xxxxxxxxxxxxxx>
Date: Wed, 13 Oct 2010 15:36:41 +0200
Cc: Ian <Ian.Campbell@xxxxxxxxxxxxx>, Jeremy Fitzhardinge <jeremy@xxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Delivery-date: Wed, 13 Oct 2010 06:37:44 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20101012164433.GB21567@xxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization: Eikelenboom IT services
References: <1452957126.20101012182813@xxxxxxxxxxxxxx> <20101012164433.GB21567@xxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
By messing a bit with printk's and debug settings a warn_on in the hypervisor 
is being triggered when starting the videograbbing domU:

mapping kernel into physical memory
about to get started...
(XEN) [2010-10-13 13:30:44] Xen WARN at msi.c:636
(XEN) [2010-10-13 13:30:44] ----[ Xen-4.1-unstable  x86_64  debug=y  Tainted:   
 C ]----
(XEN) [2010-10-13 13:30:44] CPU:    2
(XEN) [2010-10-13 13:30:44] RIP:    e008:[<ffff82c48015d797>] 
pci_enable_msi+0x48a/0x9d5
(XEN) [2010-10-13 13:30:44] RFLAGS: 0000000000010216   CONTEXT: hypervisor
(XEN) [2010-10-13 13:30:44] rax: 0000000000000004   rbx: 00000000fe5fe000   
rcx: 0000000000000001
(XEN) [2010-10-13 13:30:44] rdx: 0000000000000004   rsi: 0000000000000282   
rdi: ffff82c48024e940
(XEN) [2010-10-13 13:30:44] rbp: ffff830237e57dc8   rsp: ffff830237e57cf8   r8: 
 0000000000000009
(XEN) [2010-10-13 13:30:44] r9:  000000000000003a   r10: 0000000000000092   
r11: 0000000000000213
(XEN) [2010-10-13 13:30:44] r12: 0000000000000000   r13: ffff830237e57ea8   
r14: ffff83020211ed10
(XEN) [2010-10-13 13:30:44] r15: 0000000000000008   cr0: 000000008005003b   
cr4: 00000000000006f0
(XEN) [2010-10-13 13:30:44] cr3: 0000000225f0e000   cr2: ffff880004e93d68
(XEN) [2010-10-13 13:30:44] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 
e010   cs: e008
(XEN) [2010-10-13 13:30:44] Xen stack trace from rsp=ffff830237e57cf8:
(XEN) [2010-10-13 13:30:44]    ffff830237e57d38 ffff82c480126b66 
ffff830237e57e18 0700000000000010
(XEN) [2010-10-13 13:30:44]    0000000000001000 0000000000000030 
00000000fe5ff000 00000000fe5ff000
(XEN) [2010-10-13 13:30:44]    0000009000077d68 ffff83014601ad10 
0000000700000246 0000000000000000
(XEN) [2010-10-13 13:30:44]    0000000700000092 0000000000000000 
ffff83020211eda8 00000000000fe5ff
(XEN) [2010-10-13 13:30:44]    00000000000fe5ff ffff8301622fde28 
0000000000000202 ffff830237e57da8
(XEN) [2010-10-13 13:30:44]    ffff82c480120680 ffff830237e57ea8 
00000000ffffffed ffff830146a24000
(XEN) [2010-10-13 13:30:44]    0000000000000057 0000000000000048 
ffff830237e57e48 ffff82c48015f16e
(XEN) [2010-10-13 13:30:44]    0000000025dfc910 000000000000015c 
0000000000000048 0000000000000120
(XEN) [2010-10-13 13:30:44]    ffff830237e82480 0000000000000282 
ffff83020211ed10 ffff830237e57e28
(XEN) [2010-10-13 13:30:44]    ffff82c480120680 ffff88002df4bb30 
0000000000000057 ffff830146a24000
(XEN) [2010-10-13 13:30:44]    0000000000000048 ffff830146a24190 
ffff830237e57ef8 ffff82c480172806
(XEN) [2010-10-13 13:30:44]    0000000180196b1a ffff830237e5a020 
ffff830200000004 ffff830237e57ea8
(XEN) [2010-10-13 13:30:44]    000000000000000b ffffffffffffffff 
0000000000000007 0000000000000000
(XEN) [2010-10-13 13:30:44]    00000000fe5fe000 aaaaaaaaaaaaaaaa 
0000000000000007 0000000000000048
(XEN) [2010-10-13 13:30:44]    00000000fe5fe000 0000000000000000 
0000000000000246 ffff8300c7e88000
(XEN) [2010-10-13 13:30:44]    000000000000000b ffff8800278c4400 
0000000000000011 ffff88002ffea700
(XEN) [2010-10-13 13:30:44]    00007cfdc81a80c7 ffff82c480202a82 
ffffffff8100942a 0000000000000021
(XEN) [2010-10-13 13:30:44]    ffff88002ffea700 0000000000000011 
ffff8800278c4400 000000000000000b
(XEN) [2010-10-13 13:30:44]    ffff88002df4bbd0 00000000000006a1 
0000000000000213 ffff88002fc20200
(XEN) [2010-10-13 13:30:44]    ffffffff810df6ea 0000000000000011 
0000000000000021 ffffffff8100942a
(XEN) [2010-10-13 13:30:44] Xen call trace:
(XEN) [2010-10-13 13:30:44]    [<ffff82c48015d797>] pci_enable_msi+0x48a/0x9d5
(XEN) [2010-10-13 13:30:44]    [<ffff82c48015f16e>] map_domain_pirq+0x275/0x363
(XEN) [2010-10-13 13:30:44]    [<ffff82c480172806>] do_physdev_op+0x826/0x10b0
(XEN) [2010-10-13 13:30:44]    [<ffff82c480202a82>] syscall_enter+0xf2/0x14c
(XEN) [2010-10-13 13:30:44]
(XEN) [2010-10-13 13:30:44] SEIK bus: 7 slot: 0 func:0 msi->table_base: 
fe5fe000 read_pci_mem_bar: 4
(XEN) [2010-10-13 13:30:44] SEIK pba_paddr: 4

it's this one:
WARN_ON(msi->table_base != read_pci_mem_bar(bus, slot, func, bir));

I have added some printk's .. and read_pci_mem_bar seems to return a bogus 
value .. the pba_addr is used later in the function, but i can't oversee if and 
when this could have implications.
This also occurs when disabling the pci_resource_align on the kernel line.

lspci on dom0 shows:

07:00.0 USB Controller: NEC Corporation Device 0194 (rev 03) (prog-if 30)
        Subsystem: ASUSTeK Computer Inc. Device 8413
        Flags: bus master, fast devsel, latency 0, IRQ 33
        Memory at fe5fe000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [50] Power Management version 3
        Capabilities: [70] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 
Enable-
        Capabilities: [90] MSI-X: Enable+ Mask- TabSize=8
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff
        Capabilities: [150] #18
        Kernel driver in use: pciback


In the same function it seems to trigger
           if ( d )
            {
                /* XXX How to deal with existing mappings? */
            }

Which seems to be a bit odd for a freshly booted system with no domU restarts ?




grub menu.lst:

title           xen-4.1-unstable.gz / Debian GNU/Linux, 
2.6.32.23-xen-next-2.6.32.x-generaldebug-20101002
root            (hd0,0)
kernel          /xen-4.1-unstable.gz dom0_mem=768M loglvl=all loglvl_guest=all 
com1=115200,8n1 sync_console console_to_ring console_timestamps 
console=vga,com1 iommu=off debug lapic=debug apic_verbosity=debug apic=debug 
noirqbalance
module          /vmlinuz-2.6.32.24-xen-next-2.6.32.x-tracing-20101013 
root=/dev/mapper/serveerstertje-root ro earlyprintk=xen max_loop=255 
loop_max_part=63 libata.noacpi=1 debug loglevel=10 noirqbalance irqbalance=off 
iommu=soft xen-pciback.hide=(03:06.0)(07:00.0)(09:01.0)(09:01.1)(09:01.2) 
pci=resource_alignment=03:06.0;07:00.0;09:01.0;09:01.1;09:01.2;
module          /initrd.img-2.6.32.24-xen-next-2.6.32.x-tracing-20101013




--

Sander



Tuesday, October 12, 2010, 6:44:33 PM, you wrote:

> On Tue, Oct 12, 2010 at 06:28:13PM +0200, Sander Eikelenboom wrote:
>> Hi Keir,
>> 
>> Does xen and/or the xen console depend on physical cpu 0 ?

> Usually the console for Dom0, and I think all other domains go
> through CPU0. Let me CC Ian here, who has been mucking in this
> area and found some bugs (and produced fixes).

> Ian, that bug you found with not clearing the eventchannel - that
> wouldn't have an impact here, right?

>> 
>> I'm still trying to solve the mystery of my machine freezing when doing:
>> 
>>  - videograbbing in a domU with a usb3 pci-express controller passed through 
>> (seems to cause quite a few interrupts)
>>  - compiling a linux kernel with "make -j 6"
>> 
>> It's a 6 core AMD phenom x6.
>> 
>> Without cpu pinning:
>> I can freeze the machine easily within a minute after starting the compile, 
>> at first xen serial console also slows down under the load (slow updates).
>> When the machine freezes i can't do anything with xen serial console.
>> 
>> With cpu pinning:
>> By not using the pcpu 0 at all for any domain, and pinning the domain with 
>> the videograbber to it's own pcpu (pcpu 5)  it seems the machine keeps 
>> running after 20 "make -j6" iterations of kernel compilation.
>> Xen serial console stays responsive and doesn't slow down during the kernel 
>> compilation. The videograbber shows no problem grabbing video.
>> 

> AHA! So finally closer to the mystery.

> Can you provide the /proc/interrupts of the Dom0?

> I wonder if this is related to the isseu I had some time ago, and never got
> to look at. The problem was that during heavy compilation (this is a 2 Nehelem
> socket box, just running Dom0 - no guests), the keyboard and USB driver would
> stop getting interrupts.  So the drivers would start polling which is quite 
> slow,
> albeit servicable, and then at some point it would pick up again.

> The weirdness was that the /proc/interrupts showed absolutly _no_ interrupts 
> on CPU0
> during that time - as if Xen just forgot to update them. Jeremy suggested I 
> try to
> disable Xen IRQ balance (noirqbalance on Xen command line) in case that is 
> it, and to my
> emberrasement I haven't tried that yet.

> Did you try that? I think somebody suggested that but I can't recall whether 
> it
> was for this issue?
>> 
>> Name                                ID  VCPU   CPU State   Time(s) CPU 
>> Affinity
>> Domain-0                             0     0     3   r--    2169.7 1-4
>> Domain-0                             0     1     1   -b-    2339.3 1-4
>> Domain-0                             0     2     2   -b-    2358.9 1-4
>> Domain-0                             0     3     3   -b-    2298.2 1-4
>> Domain-0                             0     4     1   -b-    2221.9 1-4
>> Domain-0                             0     5     4   -b-    2287.7 1-4
>> backup                               9     0     4   -b-      10.6 1-4
>> database                             1     0     4   -b-      45.3 1-4
>> davical                              5     0     3   -b-       8.7 1-4
>> git                                  8     0     2   -b-       7.9 1-4
>> mail                                 2     0     4   -b-       8.0 1-4
>> samba                                3     0     3   -b-      11.1 1-4
>> security                             7     0     5   r--    1433.2 5
>> www                                  4     0     1   -b-      10.2 1-4
>> zabbix                               6     0     3   -b-      21.2 1-4
>> 
>> 
>> Is there a way a deadlock could occur between hypervisor <-> dom0 <-> domU 
>> especially related to passthrough/interrupts in the context of pcpu 0 ?

> I don't know, but I do know that the IRQ handling in Xen 4.0 changed 
> significantly compared
> to 3.4. I don't remember if you ever ran this setup under 3.4?
>> 
>> --
>> Sander



-- 
Best regards,
 Sander                            mailto:linux@xxxxxxxxxxxxxx


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel