[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] qemu-xen-dir + PCI passthrough = BOOM



Friday, January 10, 2014, 1:17:08 AM, you wrote:


> Friday, January 10, 2014, 12:54:43 AM, you wrote:


>> Thursday, January 9, 2014, 3:56:24 PM, you wrote:

>>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>>>> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>>>> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>>>> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>>>> > > > [...]
>>>> > > > > > Those Xen report something like:
>>>> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 
>>>> > > > > > >
>>>> > > > > > 131328
>>>> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
>>>> > > > > > memflags=0 (62 of 64)
>>>> > > > > > 
>>>> > > > > > ?
>>>> > > > > > 
>>>> > > > > > (I tryied to reproduce the issue by simply add many emulated 
>>>> > > > > > e1000 in
>>>> > > > > > QEMU :) )
>>>> > > > > > 
>>>> > 
>>>> > > -bash-4.1# lspci -s 01:00.0 -v 
>>>> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network 
>>>> > > Connection (rev 01)
>>>> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server 
>>>> > > Adapter
>>>> > >         Flags: fast devsel, IRQ 16
>>>> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] 
>>>> > > [size=128K]
>>>> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] 
>>>> > > [size=4M]
>>>> > >         I/O ports at e020 [disabled] [size=32]
>>>> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] 
>>>> > > [size=16K]
>>>> > >         Expansion ROM at fb400000 [disabled] [size=4M]
>>>> > 
>>>> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>>>> > allocate memory for it. Will have maybe have to find another way.
>>>> > qemu-trad those not seems to allocate memory, but I haven't been very
>>>> > far in trying to check that.
>>>> 
>>>> And indeed that is the case. The "Fix" below fixes it.
>>>> 
>>>> 
>>>> Based on that and this guest config:
>>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>>>> memory = 2048
>>>> boot="d"
>>>> maxvcpus=32
>>>> vcpus=1
>>>> serial='pty'
>>>> vnclisten="0.0.0.0"
>>>> name="latest"
>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>> pci = ["01:00.0"]
>>>> 
>>>> I can boot the guest.

>>> And can you access the ROM from the guest ?


>>> Also, I have another patch, it will initialize the PCI ROM BAR like any
>>> other BAR. In this case, if qemu is envolved in the access to ROM, it
>>> will print an error, like it the case for other BAR. 

>>> I tried to test it, but it was with an embedded VGA card. When I dump
>>> the ROM, I got the same one as the emulated card instead of the ROM from
>>> the device.

>> Ah this is what i reported earlier ..

>> If you would like it more funky .. use Dario's patches to be able to use 
>> vga="none" and use:
>> vga="none"
>> nographic=1
>> xen_platform_pci=1

>> And when you dump the rombar of the passedthrough vga card, you will end up 
>> with the iPXE rom of the emulated NIC.

>> So it is pointing at the first / last / a random rom ... but at least it 
>> doesn't seem directly tied to pointing at
>> another VGA rom (which was my first assumption some time ago).

>> When i go one step further .. by also disabling the xen platform pci device 
>> .. it doesn't boot any more.


>> Strange thing is that all the addresses in debug messages (host kernel .. 
>> hvmloader seabios qemu and guest kernel) for the rom bar seem to correspond 
>> when the translation is taken into account ..
>> so nothing obvious there ...

> Perhaps i have to correct myself here ... with:
>  vga="none"
>  nographic=1
>  xen_platform_pci=1

> There seems to be a discrepancy between what the guest kernel reports at boot 
> and the guest lspci output:
> [    0.000000] e820: [mem 0x40000000-0xfbffffff] available for PCI devices
> <snip>
> [    1.453530] PCI host bridge to bus 0000:00
> [    1.460018] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    1.466690] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
> [    1.473355] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
> [    1.480024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
> [    1.490023] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfbffffff]
> [    1.500013] pci_bus 0000:00: scanning bus
> [    1.500504] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
> [    1.500600] pci 0000:00:00.0: calling quirk_mmio_always_on+0x0/0x10
> [    1.508515] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
> [    1.515216] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
> [    1.531714] pci 0000:00:01.1: reg 0x20: [io  0xc240-0xc24f]
> [    1.540859] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
> [    1.540859] pci 0000:00:01.3: calling acpi_pm_check_blacklist+0x0/0x40
> [    1.545776] pci 0000:00:01.3: calling quirk_piix4_acpi+0x0/0x140
> [    1.545883] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 
> ACPI
> [    1.546916] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 
> SMB
> [    1.551173] pci 0000:00:01.3: calling pci_fixup_piix4_acpi+0x0/0x10
> [    1.554237] pci 0000:00:02.0: [5853:0001] type 00 class 0xff8000
> [    1.556666] pci 0000:00:02.0: reg 0x10: [io  0xc000-0xc0ff]
> [    1.566373] pci 0000:00:02.0: reg 0x14: [mem 0xf0000000-0xf0ffffff pref]
> [    1.596780] pci 0000:00:04.0: [1002:6759] type 00 class 0x030000
> [    1.656719] pci 0000:00:04.0: reg 0x10: [mem 0xe0000000-0xefffffff 64bit 
> pref]
> [    1.693355] pci 0000:00:04.0: reg 0x18: [mem 0xf1060000-0xf107ffff 64bit]
> [    1.743361] pci 0000:00:04.0: reg 0x20: [io  0xc100-0xc1ff]
> [    1.826692] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
> [    1.828722] pci 0000:00:04.0: supports D1 D2

> 00:04.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks 
> [Radeon HD 6570] (prog-if 00 [VGA controller])
>         Subsystem: PC Partner Limited Device e193
>         Physical Slot: 4
>         Flags: fast devsel, IRQ 32
>         Memory at e0000000 (64-bit, prefetchable) [size=256M]
>         Memory at f1060000 (64-bit, non-prefetchable) [size=128K]
>         I/O ports at c100 [size=256]
>         [virtual] Expansion ROM at f1000000 [disabled] [size=128K]
>         Capabilities: [50] Power Management version 3
>         Capabilities: [58] Express Legacy Endpoint, MSI 00
>         Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Capabilities: [100] #1002


> At least these 2 don't seem to match up:
> [    1.826692] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]

> [virtual] Expansion ROM at f1000000 [disabled] [size=128K]

And from xl dmesg:

(d23) [2014-01-10 00:07:47] pci dev 03:0 bar 30 size 000040000: 0f1000000

which is .. probably the NIC ... but which is mysteriously never shown in lspci 
(thought i had reported that earlier as well):

lspci -v
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
        Subsystem: Red Hat, Inc Qemu virtual machine
        Flags: bus master, fast devsel, latency 0

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
        Subsystem: Red Hat, Inc Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0

00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 
(prog-if 80 [Master])
        Subsystem: Red Hat, Inc Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0
        [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
        [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
        [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
        [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
        I/O ports at c240 [size=16]
        Kernel driver in use: PIIX_IDE

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
        Subsystem: Red Hat, Inc Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0, IRQ 9

00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
        Subsystem: XenSource, Inc. Xen Platform Device
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 24
        I/O ports at c000 [size=256]
        Memory at f0000000 (32-bit, prefetchable) [size=16M]
        Kernel driver in use: xen-platform-pci

00:04.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks 
[Radeon HD 6570] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited Device e193
        Physical Slot: 4
        Flags: fast devsel, IRQ 32
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f1060000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at c100 [size=256]
        [virtual] Expansion ROM at f1000000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] #1002


which is defined in the guest config as:
vif = [ 'bridge=xen_bridge,ip=192.168.1.44,mac=00:16:3A:C6:76:65, model=e1000'  
]


>> --
>> Sander

>>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
>>> index 6dd7a68..2bbdb6d 100644
>>> --- a/hw/xen/xen_pt.c
>>> +++ b/hw/xen/xen_pt.c
>>> @@ -440,8 +440,8 @@ static int 
>>> xen_pt_register_regions(XenPCIPassthroughState *s)
>>>  
>>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>>>  
>>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>>> -                                      "xen-pci-pt-rom", d->rom.size);
>>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
>>> +                              "xen-pci-pt-rom", d->rom.size);
>>>          pci_register_bar(&s->dev, PCI_ROM_SLOT, 
>>> PCI_BASE_ADDRESS_MEM_PREFETCH,
>>>                           &s->rom);
>>>  








_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.