Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Julien Grall <julien@xxxxxxx>]

On 26/10/2022 14:17, Rahul Singh wrote:
> Hi All,
Hi Rahul,

> At Arm, we started to implement the POC to support 2 levels of page 
> tables/nested translation in SMMUv3.
> To support nested translation for guest OS Xen needs to expose the virtual 
> IOMMU. If we passthrough the
> device to the guest that is behind an IOMMU and virtual IOMMU is enabled for 
> the guest there is a need to
> add IOMMU binding for the device in the passthrough node as per [1]. This email 
> is to get an agreement on
> how to add the IOMMU binding for guest OS.
>
> Before I will explain how to add the IOMMU binding let me give a brief overview 
> of how we will add support for virtual
> IOMMU on Arm. In order to implement virtual IOMMU Xen need SMMUv3 Nested 
> translation support. SMMUv3 hardware
> supports two stages of translation. Each stage of translation can be 
> independently enabled. An incoming address is logically
> translated from VA to IPA in stage 1, then the IPA is input to stage 2 which 
> translates the IPA to the output PA. Stage 1 is
> intended to be used by a software entity( Guest OS) to provide isolation or 
> translation to buffers within the entity, for example,
> DMA isolation within an OS. Stage 2 is intended to be available in systems 
> supporting the Virtualization Extensions and is
> intended to virtualize device DMA to guest VM address spaces. When both stage 1 
> and stage 2 are enabled, the translation
> configuration is called nesting.
>
> Stage 1 translation support is required to provide isolation between different 
> devices within the guest OS. XEN already supports
> Stage 2 translation but there is no support for Stage 1 translation for guests. 
> We will add support for guests to configure
> the Stage 1 transition via virtual IOMMU. XEN will emulate the SMMU hardware 
> and exposes the virtual SMMU to the guest.
> Guest can use the native SMMU driver to configure the stage 1 translation. When 
> the guest configures the SMMU for Stage 1,
> XEN will trap the access and configure the hardware accordingly.
>
> Now back to the question of how we can add the IOMMU binding between the 
> virtual IOMMU and the master devices so that
> guests can configure the IOMMU correctly. The solution that I am suggesting is 
> as below:
>
> For dom0, while handling the DT node(handle_node()) Xen will replace the phandle in the 
> "iommus" property with the virtual
> IOMMU node phandle.
Below, you said that each IOMMUs may have a different ID space. So 
shouldn't we expose one vIOMMU per pIOMMU? If not, how do you expect the 
user to specify the mapping?
> For domU guests, when passthrough the device to the guest as per [2],  add the 
> below property in the partial device tree
> node that is required to describe the generic device tree binding for IOMMUs 
> and their master(s)
>
> "iommus = < &magic_phandle 0xvMasterID>
>         • magic_phandle will be the phandle ( vIOMMU phandle in xl)  that will 
> be documented so that the user can set that in partial DT node (0xfdea).
Does this mean only one IOMMU will be supported in the guest?

>         • vMasterID will be the virtual master ID that the user will provide.
>
> The partial device tree will look like this:
> /dts-v1/;
>   
> / {
>      /* #*cells are here to keep DTC happy */
>      #address-cells = <2>;
>      #size-cells = <2>;
>   
>      aliases {
>          net = &mac0;
>      };
>   
>      passthrough {
>          compatible = "simple-bus";
>          ranges;
>          #address-cells = <2>;
>          #size-cells = <2>;
>          mac0: ethernet@10000000 {
>              compatible = "calxeda,hb-xgmac";
>              reg = <0 0x10000000 0 0x1000>;
>              interrupts = <0 80 4  0 81 4  0 82 4>;
>             iommus = <0xfdea 0x01>;
>          };
>      };
> };
>   
> In xl.cfg we need to define a new option to inform Xen about vMasterId to pMasterId mapping and to which IOMMU device this
> the master device is connected so that Xen can configure the right IOMMU. This 
> is required if the system has devices that have
> the same master ID but behind a different IOMMU.
In xl.cfg, we already pass the device-tree node path to passthrough. So 
Xen should already have all the information about the IOMMU and 
Master-ID. So it doesn't seem necessary for Device-Tree.
For ACPI, I would have expected the information to be found in the IOREQ.

So can you add more context why this is necessary for everyone?

>   
> iommu_devid_map = [ “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS” , “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS”]
>         • PMASTER_ID is the physical master ID of the device from the physical 
> DT.
>         • VMASTER_ID is the virtual master Id that the user will configure in 
> the partial device tree.
>         • IOMMU_BASE_ADDRESS is the base address of the physical IOMMU device 
> to which this device is connected.
Below you give an example for Platform device. How would that fit in the 
context of PCI passthrough?
>   
> Example: Let's say the user wants to assign the below physical device in DT to the guest.
>   
> iommu@4f000000 {
>                  compatible = "arm,smmu-v3";
>                 interrupts = <0x00 0xe4 0xf04>;
>                  interrupt-parent = <0x01>;
>                  #iommu-cells = <0x01>;
>                  interrupt-names = "combined";
>                  reg = <0x00 0x4f000000 0x00 0x40000>;
>                  phandle = <0xfdeb>;
>                  name = "iommu";
> };
So I guess this node will be written by Xen. How will you the case where 
there are extra property to added (e.g. dma-coherent)?
>   
> test@10000000 {
>         compatible = "viommu-test”;
>         iommus = <0xfdeb 0x10>;
I am a bit confused. Here you use 0xfdeb for the phandle but below...

>         interrupts = <0x00 0xff 0x04>;
>         reg = <0x00 0x10000000 0x00 0x1000>;
>         name = "viommu-test";
> };
>   
> The partial Device tree node will be like this:
>   
> / {
>      /* #*cells are here to keep DTC happy */
>      #address-cells = <2>;
>      #size-cells = <2>;
>   
>      passthrough {
>          compatible = "simple-bus";
>          ranges;
>          #address-cells = <2>;
>          #size-cells = <2>;
>
>         test@10000000 {
>                 compatible = "viommu-test";
>                 reg = <0 0x10000000 0 0x1000>;
>                 interrupts = <0 80 4  0 81 4  0 82 4>;
>                 iommus = <0xfdea 0x01>;
... you use 0xfdea. Does this mean 'xl' will rewrite the phandle?

>          };
>      };
> };
>   
>   iommu_devid_map = [ “0x10@0x01,0x4f000000”]
>         • 0x10 is the real physical master id from the physical DT.
>         • 0x01 is the virtual master Id that the user defines as a partial 
> device tree.
>         • 0x4f000000 is the base address of the IOMMU device.
Cheers,

--
Julien Grall