Re: [Xen-devel] GPU passthrough on ARM

[Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>]

Hi, Martin

On Fri, Jan 26, 2018 at 8:05 PM, Oleksandr Tyshchenko
<olekstysh@xxxxxxxxx> wrote:
> On Fri, Jan 26, 2018 at 3:49 PM, Julien Grall <julien.grall@xxxxxxxxxx> wrote:
>> Hi,
>>
>> I am CCing Oleksandr. He knows better than me this platform.
>
> Hi, Julien.
>
> OK, thank you, I will try to provide some pointers.
>
>>
>> Cheers,
>>
>> On 26/01/18 00:29, Martin Kelly wrote:
>>>
>>> On 01/25/2018 04:17 AM, Julien Grall wrote:
>>>>
>>>>
>>>>
>>>> On 24/01/18 22:10, Martin Kelly wrote:
>>>>>
>>>>> Hi,
>>>>
>>>>
>>>> Hello,
>>>>
>>>>> Does anyone know if GPU passthrough is supported on ARM? (e.g. for a GPU
>>>>> integrated into an ARM SoC). I checked documentation and the code, but I
>>>>> couldn't tell for sure.
>>>>>
>>>>> If so, what are the hardware requirements for it? If not, is it feasible
>>>>> to do in the future?
>>>>
>>>>
>>>> Xen Arm supports device integrated into an ARM SoC. In general we highly
>>>> recommend to have the GPU behind an IOMMU. So passthrough would be fully
>>>> secure.
>>>>
>>>> Does your platform has an IOMMU? If so which one? Do you know if the GPU
>>>> is behind it?
>>>>
>>>> It would be possible to do passthrough without IOMMU, but that's more
>>>> complex and would require some hack in Xen to make sure the guest memory is
>>>> direct mapped (e.g guest physical address = host physical address).
>>>>
>>>> For more documentation on how to do it (see [1] and [2]).
>>>>
>>>> Cheers,
>>>>
>>>> [1]
>>>> https://events.static.linuxfound.org/sites/events/files/slides/talk_5.pdf
>>>> [2] https://wiki.xen.org/images/1/17/Device_passthrough_xen.pdf
>>>>
>>>
>>> Hi Julien,
>>>
>>> Thanks very much for the information. I'm looking at the Renesas R-Car H3
>>> R8A7795, which has an IOMMU (using the Linux ipmmu-vmsa driver in
>>> drivers/iommu/ipmmu-vmsa.c). Looking at the device tree for it
>>> (r8a7795.dtsi), it appears you could pass through the display@feb00000 node
>>> for the DRM driver.
>>>
>>> I did notice this patch series, which didn't get merged:
>>>
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg02679.html
>>>
>>> Presumably that driver would be needed in Xen.
>>>
>>> Are there any gotchas I'm missing? Is GPU passthrough on ARM something
>>> that is "theoretically doable" or something that has been done already and
>>> shown to be performant?

I assume the H3 SoC version you are using is ES2.0, because of r8a7795.dtsi.

BTW, what BSP version are you using? I am wondering what is your use-case?
If you want to keep GPU in some dedicated domain without no hardware
at all, you have to use something like PV DRM frontend running here
and PV DRM backend in the hardware/driver domain.
The things are going to be much simple if you pass through all
required display sub-components as well, for the "rcar-du" DRM to be
functional.
Which way are you looking for?

Anyway, in both cases you have to pass through GPU. For that some
activities should be done:

1. Xen side:

As for the patch series, you are right, you have to base on it. There
are two separate patch series which haven't upstreamed yet,
but needed for the passthrough feature to work on R-Car Gen3 SoCs (M3, H3).

https://www.mail-archive.com/xen-devel@xxxxxxxxxxxxx/msg115901.html
https://www.mail-archive.com/xen-devel@xxxxxxxxxxxxx/msg116038.html

Also additional patch is needed to teach IPMMU-VMSA driver to handle
devices which are hooked up to multiple IPMMU caches.
Since the GPU on H3 SoC is connected to multiple IPMMU caches: PV0 - PV3.

I have created new branch you can simply base on to get required
support in hand.
repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_next

2. Device trees and guest config file:

2.1. You have to add following to the domain 0 device tree:

There is no magic here. This is just to enable corresponding IPMMUs,
hooked up GPU to them and notify Xen
that device is going to be pass throughed.

&gsx {
    xen,passthrough;

    iommus = <&ipmmu_pv0 0>, <&ipmmu_pv0 1>,
    <&ipmmu_pv0 2>, <&ipmmu_pv0 3>,
    <&ipmmu_pv0 4>, <&ipmmu_pv0 5>,
    <&ipmmu_pv0 6>, <&ipmmu_pv0 7>,
    <&ipmmu_pv1 0>, <&ipmmu_pv1 1>,
    <&ipmmu_pv1 2>, <&ipmmu_pv1 3>,
    <&ipmmu_pv1 4>, <&ipmmu_pv1 5>,
    <&ipmmu_pv1 6>, <&ipmmu_pv1 7>,
    <&ipmmu_pv2 0>, <&ipmmu_pv2 1>,
    <&ipmmu_pv2 2>, <&ipmmu_pv2 3>,
    <&ipmmu_pv2 4>, <&ipmmu_pv2 5>,
    <&ipmmu_pv2 6>, <&ipmmu_pv2 7>,
    <&ipmmu_pv3 0>, <&ipmmu_pv3 1>,
    <&ipmmu_pv3 2>, <&ipmmu_pv3 3>,
    <&ipmmu_pv3 4>, <&ipmmu_pv3 5>,
    <&ipmmu_pv3 6>, <&ipmmu_pv3 7>;
};

 &ipmmu_pv0 {
     status = "okay";
 };

 &ipmmu_pv1 {
     status = "okay";
 };

 &ipmmu_pv2 {
     status = "okay";
 };

 &ipmmu_pv3 {
     status = "okay";
 };

 &ipmmu_mm {
     status = "okay";
 };

2.2. You have to add following to the guest config file:

I might mistake here, please use existing documentation to see how a
guest config file
should be properly modified. For example
https://events.static.linuxfound.org/sites/events/files/slides/talk_5.pdf

dtdev = [ "/soc/gsx@fd000000" ]

irqs = [ 151 ]

iomem = [ "fd000,40" ]

device_tree = "domU.dtb" <- This is the guest partial device tree.

2.3 Actually the guest partial device tree I have used is:

/dts-v1/;

#include <dt-bindings/interrupt-controller/arm-gic.h>

/ {
    #address-cells = <2>;
    #size-cells = <2>;

    passthrough {
        compatible = "simple-bus";
        ranges;

        #address-cells = <2>;
        #size-cells = <2>;

        gsx: gsx@fd000000 {
            compatible = "renesas,gsx";
            reg = <0 0xfd000000 0 0x3ffff>;
            interrupts = <GIC_SPI 119 IRQ_TYPE_LEVEL_HIGH>;
            /*clocks = <&cpg CPG_MOD 112>;*/
            /*power-domains = <&sysc R8A7795_PD_3DG_E>;*/
        };
    };
};

Hope this helps.

>>>
>>> Thanks again,
>>> Martin
>>
>>
>> --
>> Julien Grall
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxxx
>> https://lists.xenproject.org/mailman/listinfo/xen-devel
>
>
>
> --
> Regards,
>
> Oleksandr Tyshchenko

-- 
Regards,

Oleksandr Tyshchenko

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel