Re: [PATCH v4 0/7] SMMU handling for PCIe Passthrough on ARM

[Stewart Hildebrand <stewart.hildebrand@xxxxxxx>]

On 6/7/23 03:19, Julien Grall wrote:
> Hi Stewart,
> 
> On 07/06/2023 04:02, Stewart Hildebrand wrote:
>> This series introduces SMMU handling for PCIe passthrough on ARM. These 
>> patches
>> are independent from (and don't depend on) the vPCI reference 
>> counting/locking
>> work in progress, and should be able to be upstreamed independently.
> 
> Can you clarify how this code was tested? Does this require code not yet
> upstreamed?

I'm testing the series standalone (+ config changes) by using a PCI device in 
dom0, and also in combination with the vPCI series [3] [4] for passthrough to a 
domU.


Here are some more details on the test cases I'm using.


1. Using the PCI device in dom0 with the pci-passthrough=yes arg. In this case 
a couple of additional config changes [1] [2] are needed to select 
CONFIG_HAS_PCI=y, CONFIG_HAS_VPCI=y, and make has_vpci() return true. Aside 
from this series itself and the config changes, nothing else not-yet-upstreamed 
is required for this test case. It is on my TODO list to upstream these config 
changes, which I think will be useful on their own, not necessarily as part of 
any other series.

Actually, your question prompted me to look at my test cases a bit more 
carefully, and I discovered a case that v4 of this series doesn't handle right. 
In order for the PCI device to be usable in dom0, it should be assigned by 
default to dom0/hardware domain during PHYSDEVOP_pci_device_add. But v4 of this 
series doesn't assign it by default to dom0/hardware domain. I initially missed 
this because I happened to assign the stream ID of the PCI device to dom0 by 
the iommus property.

In other words, I initially tested with this:

&pcie {
    iommus = <&smmu 0x4d0>;
    iommu-map = <0x0 &smmu 0x4d0 0x10000>;
    iommu-map-mask = <0x0>;
};

But I should be testing with this (i.e. omitting the iommus property):

&pcie {
    iommu-map = <0x0 &smmu 0x4d0 0x10000>;
    iommu-map-mask = <0x0>;
};

Omitting iommus currently breaks using a PCI device in dom0 in v4. I'll fix 
this in v5.


2. To test the phantom bits, since I don't have a phantom device readily 
available, I use the pci-phantom arg and a carefully constructed iommu-map 
property. The PCI device's stream ID is actually 0x4d0, but I put 0x4ce in the 
iommu-map to make it appear as if it's one of the phantom functions generating 
the DMA traffic.

pci-phantom=01:00,1

&pcie {
    /* phantom test */
    iommu-map = <0x0 &smmu 0x4ce 0x10000>;
    iommu-map-mask = <0x7>;
};


3. Passthrough to a domU. In this case the vPCI series is needed [3], and an 
MSI series not yet submitted to the list [4] (or another way to hack/assign the 
IRQ to the domU), in addition to the 2 config changes [1] [2]. The procedure is 
described at [5].

[1] 
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commit/9a08f1f7ce28ec619640ba9ce11018bf443e9a0e
[2] 
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commit/27be1729ce8128dbe37275ce7946b2fbd2e5a382
[3] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg00863.html
[4] 
https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commits/poc/pci-passthrough
[5] https://lists.xenproject.org/archives/html/xen-devel/2022-12/msg00682.html