Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

["Zhang, Yang Z" <yang.z.zhang@xxxxxxxxx>]

Jan Beulich wrote on 2013-09-11:
>>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@xxxxxxxxxx> wrote:
>> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
>> <JBeulich@xxxxxxxx>
>>  wrote:
>>>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@xxxxxxxxxx> wrote:
>>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
>>>> <JBeulich@xxxxxxxx>
>>>>  wrote:
>>>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@xxxxxxxxxx> wrote:
>>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>>>>  
>>>>>>  I'll try adding one of my LSI cards and see the comparative
>>>>>> behaviour. Right now I don't even know if the phantom device  is
>>>>>> on the SAS card or the motherboard.
>>>>> 
>>>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>>>> likely that this other device also is on that card.
>>>>> 
>>>>> I guess the issue is mainly because the device itself is a PCI
>>>>> one, while the immediately upstream bridge (where I mean only the
>>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>>>> them. And as long as firmware doesn't know about that bridge and
>>>>> the bridge doesn't properly handle config space accesses to it,
>>>>> such a device just can't be used with an IOMMU (without some yet
>>>>> to be invented workaround).
>>>>> 
>>>>  I'm actually thinking about Konrad's proposed hack in that
>>>> thread from 3 years ago. If the device IDs are parameterized  out
>>>> rather than hard-coded, then this could work in nearly the  same
>>>> was as xen-pciback in terms of usage. Pass the phantom  device IDs
>>>> as parameters to the module. Done that way it  might even be
>>>> considered clean enough to be fit for public  consumption.
>>> 
>>> Except that, short of being able to determine it via config space
>>> reads, we also need the resulting command line option to tell us
>>> that what kind of device that is.
>>> 
>>  Not sure I follow. Why do we need to know the device type?
> 
> Just look at set_msi_source_id() as well as
> domain_context_{mapping,unmap}() (just the most prominent
> examples): Behavior here heavily depends on the type of the device
> itself _and_ that of the upstream bridge(s).
Looks like there are many devices are failed to work. I wonder whether the 
PCI/PCIe specification tells how to detect the hidden device behind those 
devices (Like detection of phantom device). If not, I think those devices are 
buggy. Or we can say those devices are not really PCI/PCIe compatible. Since 
VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device 
failed to work under VT-d.

As Jan's suggestion, we need the user to tell us whether there is a hidden 
device or BDF behind anther device that the OS is unaware. We need to pass that 
info to Xen before pass-thought the device.

> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel


Best regards,
Yang


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel