Xen project Mailing List

[Xen-devel] PCI passthrough (pci-attach) to HVM guests bug (BAR64 addresses are bogus)

To: xen-devel@xxxxxxxxxxxxxxxxxxxx, jbeulich@xxxxxxxx

From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Date: Mon, 10 Nov 2014 12:32:48 -0500

Delivery-date: Mon, 10 Nov 2014 17:33:06 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

Hey, With Xen 4.5 (today's staging), when I boot a guest and then do pci-attach the BARs values are corrupt. For example, with this guest config: kernel="hvmloader" builder="hvm" serial="pty" memory = 2048 name = "XTT" usb=1 usbdevice='tablet' vcpus=2 vga="stdvga" vif = [ 'mac=00:0f:4b:00:00:63,bridge=xenbr0' ] disk= ['file:/root/root_image.iso,hdc:cdrom,r'] vnc=1 vnclisten="0.0.0.0" boot = "dc" And with this PCI card: 5:00.0 VGA compatible controller: NVIDIA Corporation GF104GLM [Quadro 4000M] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 34fc Flags: fast devsel, IRQ 20 Memory at fc000000 (32-bit, non-prefetchable) [size=32M] Memory at d0000000 (64-bit, prefetchable) [size=128M] Memory at d8000000 (64-bit, prefetchable) [size=64M] which in dom0 is reported as: 4064] pci 0000:05:00.0: [10de:0e3b] type 00 class 0x030000 [ 22.834087] pci 0000:05:00.0: reg 0x10: [mem 0xfc000000-0xfdffffff] [ 22.834109] pci 0000:05:00.0: reg 0x14: [mem 0xd0000000-0xd7ffffff 64bit pref] [ 22.834131] pci 0000:05:00.0: reg 0x1c: [mem 0xd8000000-0xdbffffff 64bit pref] [ 22.834144] pci 0000:05:00.0: reg 0x24: [io 0xc000-0xc07f] [ 22.834157] pci 0000:05:00.0: reg 0x30: [mem 0xfe000000-0xfe07ffff pref] When I assign said card (xl pci-attach XTT 05:00.0) the guest gives me: ... 498525] pci 0000:00:04.0: [10de:0e3b] type 00 class 0x030000 [ 152.508612] pci 0000:00:04.0: reg 0x10: [mem 0x00000000-0x01ffffff] [ 152.518320] pci 0000:00:04.0: reg 0x14: [mem 0x00000000-0x07ffffff 64bit preff ] [ 152.529301] pci 0000:00:04.0: reg 0x1c: [mem 0x00000000-0x03ffffff 64bit preff ] [ 152.540095] pci 0000:00:04.0: reg 0x24: [io 0x0000-0x007f] [ 152.548497] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 152.561018] vgaarb: device added: PCI:0000:00:04.0,decodes=io+mem,owns=none,llocks=none [ 152.572965] pci 0000:00:04.0: BAR 1: no space for [mem size 0x08000000 64bit pref] [ 152.583917] pci 0000:00:04.0: BAR 1: failed to assign [mem size 0x08000000 64 bit pref] [ 152.595528] pci 0000:00:04.0: BAR 3: assigned [mem 0xf4000000-0xf7ffffff 64bi t pref] If I boot the guest with: pci=["05:00.0"] it works fine: # dmesg | grep 05.0 [ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 [ 0.905008] pci 0000:00:03.0: reg 0x10: [mem 0xef000000-0xefffffff pref] [ 0.944101] pci 0000:00:05.0: [10de:0e3b] type 00 class 0x030000 [ 0.953013] pci 0000:00:05.0: reg 0x10: [mem 0xec000000-0xedffffff] [ 0.967016] pci 0000:00:05.0: reg 0x14: [mem 0xe0000000-0xe7ffffff 64bit pref] [ 0.973016] pci 0000:00:05.0: reg 0x1c: [mem 0xe8000000-0xebffffff 64bit pref] [ 0.981015] pci 0000:00:05.0: reg 0x24: [io 0xc200-0xc27f] [ 0.988016] pci 0000:00:05.0: reg 0x30: [mem 0xf0000000-0xf007ffff pref] [ 0.995083] vgaarb: device added: PCI:0000:00:05.0,decodes=io+mem,owns=io+mem,locks=none [ 0.997000] vgaarb: bridge control possible 0000:00:05.0 [ 3.952023] nouveau [ DEVICE][0000:00:05.0] BOOT0 : 0x0c4d80a1 [ 3.952025] nouveau [ DEVICE][0000:00:05.0] Chipset: GF104 (NVC4) [ 3.952027] nouveau [ DEVICE][0000:00:05.0] Family : NVC0 [ 3.952072] nouveau [ VBIOS][0000:00:05.0] checking PRAMIN for image... [ 3.952079] nouveau [ VBIOS][0000:00:05.0] ... signature not found [ 3.952080] nouveau [ VBIOS][0000:00:05.0] checking PROM for image... [ 4.096446] nouveau [ VBIOS][0000:00:05.0] ... appears to be valid [ 4.104012] nouveau [ VBIOS][0000:00:05.0] using image from PROM [ 4.111214] nouveau [ VBIOS][0000:00:05.0] BIT signature found [ 4.118054] nouveau [ VBIOS][0000:00:05.0] version 70.04.13.00.01 [ 4.125248] nouveau [ DEVINIT][0000:00:05.0] adaptor not initialised [ 4.131629] nouveau [ VBIOS][0000:00:05.0] running init tables [ 4.227827] nouveau [ PMC][0000:00:05.0] MSI interrupts enabled [ 4.234296] nouveau [ PFB][0000:00:05.0] RAM type: GDDR5 [ 4.240176] nouveau [ PFB][0000:00:05.0] RAM size: 1024 MiB [ 4.245878] nouveau [ PFB][0000:00:05.0] ZCOMP: 0 tags [ 4.255523] nouveau [ VOLT][0000:00:05.0] GPU voltage: 875000uv [ 4.291146] nouveau [ PTHERM][0000:00:05.0] FAN control: PWM [ 4.296582] nouveau [ PTHERM][0000:00:05.0] fan management: automatic [ 4.302668] nouveau [ PTHERM][0000:00:05.0] internal sensor: yes [ 4.309465] nouveau [ CLK][0000:00:05.0] 03: core 50 MHz memory 135 MHz [ 4.317854] nouveau [ CLK][0000:00:05.0] 07: core 405 MHz memory 324 MHz [ 4.326655] nouveau [ CLK][0000:00:05.0] 0c: core 405 MHz memory 1800 MHz [ 4.333742] nouveau [ CLK][0000:00:05.0] 0f: core 715 MHz memory 1800 MHz [ 4.341362] nouveau [ CLK][0000:00:05.0] --: core 50 MHz memory 135 MHz [ 4.749977] nouveau 0000:00:05.0: fb0: nouveaufb frame buffer device [ 4.756172] nouveau 0000:00:05.0: registered panic notifier [ 4.765041] [drm] Initialized nouveau 1.2.0 20120801 for 0000:00:05.0 on minor 0 # lspci -s 00:05.0 -v 00:05.0 VGA compatible controller: nVidia Corporation Device 0e3b (rev a1) (prog-if 00 [VGA controller]) Subsystem: Giga-byte Technology Device 34fc Physical Slot: 5 Flags: bus master, fast devsel, latency 0, IRQ 67 Memory at ec000000 (32-bit, non-prefetchable) [size=32M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=64M] I/O ports at c200 [size=128] Expansion ROM at f0000000 [disabled] [size=512K] Interesting observation: a) It does NOT make a different if I use qemu-traditinal or qemu-xen. In both cases I get the same BAR bogus numbers. b) qemu-xen logging is not that great. When I used qemu-trad I discovered: m-command: hot insert pass-through pci dev register_real_device: Assigning real physical device 05:00.0 ... register_real_device: Disable MSI translation via per device option register_real_device: Disable power management pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x5:0x0.0x0 pt_register_regions: IO region registered (size=0x02000000 base_addr=0xfc000000) pt_register_regions: IO region registered (size=0x08000000 base_addr=0xd000000c) pt_register_regions: IO region registered (size=0x04000000 base_addr=0xd800000c) pt_register_regions: IO region registered (size=0x00000080 base_addr=0x0000c001) pt_register_regions: Expansion ROM registered (size=0x00080000 base_addr=0xfe000000) pci_intx: intx=1 register_real_device: Real physical device 05:00.0 registered successfuly! IRQ type = INTx generate a sci for PHP. deassert due to disable GPE bit. ACPI:debug: write addr=0xb044, val=0x20. ACPI:debug: write addr=0xb045, val=0x0. ACPI:debug: write addr=0xb044, val=0x20. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x21. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x22. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x23. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x24. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x25. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x26. ACPI:debug: write addr=0xb045, val=0x89. ACPI:debug: write addr=0xb044, val=0x27. ACPI:debug: write addr=0xb045, val=0x89. pt_iomem_map: e_phys=f2000000 maddr=fc000000 type=0 len=33554432 index=0 first_map=1 pt_iomem_map: e_phys=f4000000 maddr=d8000000 type=8 len=67108864 index=3 first_map=1 pt_ioport_map: e_phys=1000 pio_base=c000 len=128 index=5 first_map=1 pt_msgctrl_reg_write: setup msi for dev 20 pt_msi_setup: pt_msi_setup requested pirq = 87 pt_msi_setup: msi mapped with pirq 57 pt_msi_update: Update msi with pirq 57 gvec 0 gflags 3057 pci_intx: intx=1 pt_msi_disable: Unbind msi with pirq 57, gvec 0 pt_msi_disable: Unmap msi with pirq 57 Ideas which commit id I ought to look at? _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.