Re: [Xen-devel] [RFC + Queries] Flow of PCI passthrough in ARM

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

On Mon, 6 Oct 2014, manish jaggi wrote:
> Below is the Design Flow:
> 
> =====
> Xen PCI Passthrough support for ARM
> 
> ----------------------------------------------
> 
> 
> Nomenclature:
> 
> SBDF - segment:bus:device.function.
> Segment - Number of the PCIe RootComplex. (this is also referred to as
> domain in linux terminology Domain:BDF)
> domU sbdf - refers to the sbdf of the device assigned when enumerated
> in domU. This is different from the actual sbdf.
> 
> 
> What is the requirement
> 
> 1. Any PCI device be assigned to any domU at runtime.
> 2. The assignment should not be static, system admin must be able to
> assign a PCI BDF at will.
> 3. SMMU should be programmed for the device/domain pair
> 4. MSI should be directly injected into domU.
> 5. Access to GIC should be trapped in Xen to program MSI(LPI) in ITS
> 6. FrontEnd - Backend communication between DomU-Dom0 must be limited
> to PCI config read writes.
> 
> What is support today
> 
> 1. Non PCI passthrough using a device tree. A device tree node can be
> passthrough to a domU.
> 2. SMMU is programmed (devices' stream ID is allowed to access domU
> guest memory)
> 3. By default a device is assigned to dom0 if not done a pci-hide.
> (SMMU for that device is programmed for dom0)
> 
> 
> Proposed Flow:
> 
> 1. Xen parses its device tree to find the pci nodes. There nodes
> represent the PCIe Root Complexes
> 
> 2. Xen parses its device tree to find smmu nodes which have mmu-master
> property which should point to the PCIe RCs (pci nodes)
> Note: The mapping between the pci node and the smmu node is based on:
> which smmu which translates r/w requests from that pcie
> 
> 3. All the pci nodes are assigned to the dom0.
> 4. dom0 boots
> 
> 5. dom0 enumerates all the PCI devices and calls xen hypercall
> PHYSDEVOP_pci_device_add. This in-turn programs the smmu
> Note: Also The MMIO regions for the device are assigned to the dom0.
> 
> 6. dom0 boots domU.
> Note: in domU, the pcifront bus has a msi-controller attached. It is
> assumed that the domU device tree contains the 'gicv3-its' node.
> 
> 7. dom0 calls xl pci-assignable-add <sbdf>
> 
> 8. dom0 calls xl pci-attach <domU_id> '<sbdf>,permissive=1'
> Note: XEN_DOMCTL_assign_device is called. Implemented assign_device in smmu.c
> The MMIO regions for the device are assigned to the respective domU.
> 
> 9. Front end driver is notified by xenwatch daemon and it starts
> enumerating devices.
> Note: the check of initial_domain in register_xen_pci_notifier() is removed.
> 
> 
> 10. The device driver requests the msi using pcie_enable_msi, which in
> turn calls the its driver which tries to issue commands like MAPD,
> MAPVI. which are trapped by the Xen ITS emulation driver.
> 10a. The MAPD command contains a device id, which is a stream ID
> constructed using the sbdf. The sbdf is specific to domU for that
> device.
> 10b. Xen ITS driver has to replace the domU sbdf to the actual sdbf
> and program the command in the command queue.
> 
> Now there is a _problem_ that xen does not know the mapping between
> domU streamid (sbdf) and actual sbdf.
> For ex if two pci devices are assigned to domU, xen does not know that
> which domU sbdf maps to which pci device
> Thus the Xen ITS driver fails to program the MAPD command in command
> queue, which results LPIs not programmed for that device
> 
> At the time xl pci-attach is called the domU sbdf of the device is not
> known, as the enumeration of the PCI device in domU has not started by
> that time.
> in xl pci-attach a virtual slot number can be provided but it is not
> used in ITS commands in domU.
> 
> If it is known somehow (__we need help on this__) then dom0 could
> issue a hypercall to xen to map domU sbdf to actual in the following
> format
> 
> PHYSDEVOP_map_domU_sbdf{
> 
> actual sBDF,
> domU enumerated sBDF
> and domU_ID.
> 
> }

Now the problem is much much clearer, thank you!

Actually the xen-pcifront driver in the guest knows the real PCI sbdf
for the assigned device, not just the virtual slot. On x86 xen-pcifront
makes an hypercall to enable msi/msix on the device passing the real
sbdf as argument:

drivers/pci/xen-pcifront.c:pci_frontend_enable_msix

Could we use the same hypercall to enable msi/msix on ARM? That would be
ideal.

Otherwise xen-pcifront could call a new hypercall to let Xen know the
virtual sbdf to sbdf mapping. But I would prefer not to introduce a new
hypercall and reuse the existing one.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel