[Xen-devel] RE: pci pass-through failure on xen 3.3.0

Hi Masroor,

There are many important bug fixes in the latest 3.3 tree. Maybe it's better to try the latest 3.3 code. For instance, you may need a small changeset 18381:24083a2dba7f to fix a format typo here. :-)

For the co-assignment, if a device(including multi-function device) hasn't a proper standard FLR methods, we have to use the SecondaryBusReset as a FLR method, so we require the co-assignment.

You can refer to http://xenbits.xensource.com/xen-3.3-testing.hg?rev/e61978c24d84 for details.

Thanks,

-- Dexuan

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Cui, Dexuan
Sent: 2008年10月10日 10:01
To: 'Masroor Vettuparambil'; 'xen-devel@xxxxxxxxxxxxxxxxxxx'
Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0

Hi Masroor,

Thanks for your testing! Glad to see an incorrect assumption is fixed here.

For the non-page aligned BAR issue, it's expected:

Your NIC has a non-page aligned BAR:

Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K]

Currently Xen disallows this kind of assignment due to some known issues...

What's lucky is:

Recently Yuji Shimada has posted a patch to resolve the case.

Can you try the latest Dom0(changeset 693:2b5cc22ab406)?

Thanks,

-- Dexuan

From: Masroor Vettuparambil [mailto:Masroor.Vettuparambil@xxxxxxxxxxxx]
Sent: 2008年10月10日 9:43
To: Cui, Dexuan; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: pci pass-through failure on xen 3.3.0

Hi Dexuan,

Thanks for your patch. It looks like the below problem fixed.

But now I ended up with “non-page aligned bar address” error.

Xend log and lspci config space are given below,

By the way, I could see in the script that all the pci devices behind the bridge should be co-assigned to the same guest.

Does it mean that we could not export each device function on a multi-function card to separate guests?

Xend log is,

[2008-10-09 18:07:16 7867] ERROR (xmlrpclib2:178) Internal error handling xend.domain.device_configure

Traceback (most recent call last):

File "//usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch

response = self._dispatch(method, params)

File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch

return func(*params)

File "<string>", line 1, in <lambda>

File "//usr/lib64/python/xen/xend/server/XMLRPCServer.py", line 55, in dispatch

return getattr(info, fn)(*args)

File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 821, in device_configure

return self.pci_device_configure(dev_sxp)

File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 722, in pci_device_configure

self.device_create(dev_sxp)

File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 637, in device_create

self._createDevice(dev_type, dev_config_dict)

File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1869, in _createDevice

return self.getDeviceController(deviceClass).createDevice(devConfig)

File "//usr/lib64/python/xen/xend/server/DevController.py", line 94, in createDevice

self.setupDevice(config)

File "//usr/lib64/python/xen/xend/server/pciif.py", line 431, in setupDevice

self.setupOneDevice(domain, bus, slot, func)

File "//usr/lib64/python/xen/xend/server/pciif.py", line 289, in setupOneDevice

raise VmError("pci: %: non-page-aligned MMIO BAR found." % dev.name)

ValueError: unsupported format character \047:\047 (0x3a) at index 6

Config space dump of the NIC is,

linux-242:# lspci -d 17d5: -v -xxx

08:01.0 Ethernet controller: S2io Inc. Xframe II 10Gbps Ethernet (rev 02)

Subsystem: Hewlett-Packard Company Unknown device 1337

Flags: 66MHz, medium devsel, IRQ 9

Memory at ff5f8000 (64-bit, prefetchable) [disabled] [size=32K]

Memory at ff400000 (64-bit, prefetchable) [disabled] [size=1M]

Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K]

Expansion ROM at fea00000 [disabled] [size=1M]

Capabilities: [40] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/5 Enable-

Capabilities: [60] PCI-X non-bridge device

Capabilities: [80] Vital Product Data

Capabilities: [90] MSI-X: Enable- Mask- TabSize=64

Capabilities: [a0] Power Management version 3

00: d5 17 32 58 00 00 30 02 02 00 00 02 10 40 00 00

10: 0c 80 5f ff 00 00 00 00 0c 00 40 ff 00 00 00 00

20: 0c 78 5f ff 00 00 00 00 00 00 00 00 3c 10 37 13

30: 00 00 a0 fe 40 00 00 00 00 00 00 00 09 01 ff 01

40: 05 60 8a 01 00 00 00 00 00 00 00 00 00 00 00 00

50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

60: 07 80 73 10 08 08 e3 5f 00 00 00 00 00 00 00 00

70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

80: 03 90 fc 80 00 00 00 78 00 00 00 00 00 00 00 00

90: 11 a0 3f 00 04 00 00 00 04 04 00 00 00 00 00 00

a0: 01 00 23 7e 00 20 00 dc 00 00 00 00 00 00 00 00

b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Thanks,

Masroor

From: Cui, Dexuan [mailto:dexuan.cui@xxxxxxxxx]
Sent: Wednesday, October 08, 2008 2:31 AM
To: Masroor Vettuparambil; 'xen-devel@xxxxxxxxxxxxxxxxxxx'
Subject: RE: pci pass-through failure on xen 3.3.0

Hi Masroor,

BTW, you can simply apply the patch directly: login the target host, cd /usr/lib64/python/xen, patch -p4 < fix_find_parent.patch; xend restart.

Thanks,

-- Dexuan

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Cui, Dexuan
Sent: 2008年10月8日 17:16
To: 'Masroor Vettuparambil'; 'xen-devel@xxxxxxxxxxxxxxxxxxx'
Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0

Hi Masroor,

Looks there is an incorrect assumption.

Could you please help to test the attached patch? If it doesn't work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"?

Thanks!

-- Dexuan

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Masroor Vettuparambil
Sent: 2008年10月8日 8:33
To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] pci pass-through failure on xen 3.3.0

Hi all,

I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC.

Using late binding, the pci slot 0000:08:01.0 is assigned to pciback.

But pci related xm commands are not working.

For eg:

linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices

Unexpected error: <type 'exceptions.OSError'>

Please report to xen-devel@xxxxxxxxxxxxxxxxxxx

Traceback (most recent call last):

File "/usr/sbin/xm", line 10, in <module>

main.main(sys.argv)

File "//usr/lib64/python/xen/xm/main.py", line 2883, in main

_, rc = _run_cmd(cmd, cmd_name, args)

File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd

return True, cmd(args)

File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices

devs_list = check_FLR_capability(dev_list)

File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability

coassigned_pci_list = dev.find_coassigned_devices(True)

File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices

dev = self.find_the_uppermost_pci_bridge()

File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge

dev_parent = PciDevice(dom, b, d, f)

File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__

self.detect_dev_info()

File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info

class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE)

File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16

fd = os.open(self.cfg_space_path, os.O_RDONLY)

OSError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:06:00.0/config'

I checked the above python scripts.

The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0

When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error.

Lspci tree is,

linux-242:/home/xen-3.3.0 # lspci -t

-+-[0000:06]-+-01.0-[0000:08]----01.0

| +-01.1

| +-02.0-[0000:07]--

| \-02.1

\-[0000:00]-+-00.0

+-01.0

+-01.1

+-06.0

+-07.0

+-08.0

+-09.0-[0000:05]----07.0

+-0b.0-[0000:04]----00.0

+-0c.0-[0000:03]----00.0

+-0d.0-[0000:02]--

+-0e.0-[0000:01]--

+-18.0

+-18.1

+-18.2

+-18.3

+-19.0

+-19.1

+-19.2

\-19.3

Thanks,

Masroor

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

WARNING - OLD ARCHIVES

xen-devel

[Xen-devel] RE: pci pass-through failure on xen 3.3.0