Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver

[Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>]

Hi, Rahul!

On 11/6/20 3:58 PM, Rahul Singh wrote:
> Hello Oleksandr,
>
>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko 
>> <Oleksandr_Andrushchenko@xxxxxxxx> wrote:
>>
>> Hello, Rahul!
>>
>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>> Hello Oleksandr,
>>>
>>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko 
>>>> <Oleksandr_Andrushchenko@xxxxxxxx> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>>> Hi,
>>>>>
>>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@xxxxxxxxx> 
>>>>>> wrote:
>>>>>>
>>>>>> Hi, Julien!
>>>>>>
>>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>>> Hi Oleksandr,
>>>>>>>
>>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>>
>>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux 
>>>>>>>>> driver
>>>>>>>>>       that supports both Stage-1 and Stage-2 translations.
>>>>>>>> First of all thank you for the efforts!
>>>>>>>>
>>>>>>>> I tried the patch with QEMU and would like to know if my understanding 
>>>>>>>> correct
>>>>>>>>
>>>>>>>> that this combination will not work as of now:
>>>>>>>>
>>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 
>>>>>>>> 0x00000000b8469000
>>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>>
>>>>>>>> [snip]
>>>>>>>>
>>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>>
>>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on 
>>>>>>>> QEMU side.
>>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>>> Exactly
>>>>>>> You might be able to use the stage-1 page-tables to isolate each device 
>>>>>>> in Xen. However, I don't think you will be able to share the P2M 
>>>>>>> because the page-tables layout between stage-1 and stage-2 is different.
>>>>>> So, it is even more work then
>>>>> Overall it would make more sense to spend some time adding proper support 
>>>>> in Qemu then trying to modify the driver to support Qemu right now.
>>>>>
>>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI 
>>>>>>>> passthrough
>>>>>>>>
>>>>>>>> implementation, so it could allow testing different setups and 
>>>>>>>> configurations with QEMU.
>>>>>>> I would recommend to get the SMMU supporting supporting stage-2 
>>>>>>> page-tables.
>>>>>> You mean in QEMU?
>>>>> See before.
>>>>>
>>>>>>> Regardless that, I think Xen should be able to say the SMMU is not 
>>>>>>> supported rather than crashing.
>>>>>> Yes, that would be nice
>>>>> Fully agree and we will look into that.
>>>>>
>>>>> Anything you could share so that we could quickly reproduce your setup 
>>>>> would be more then great.
>>>> Nothing special,
>>>>
>>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine 
>>>> virt,gic-version=2 \
>>>>
>>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic 
>>>> user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>>
>>>> -nographic -serial mon:stdio [..snip..]
>>>>
>>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and 
>>> XEN is able to say SMMU translation is not supported. As XEN supports 
>>> Stage-2 translation and QEMU supports Stage-1 only.
>>>
>>>
>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration 
>>> (false)
>>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>>> (XEN) I/O virtualisation disabled
>>>
>>> Only difference I observed is that you have to add option "-machine 
>>> virt,iommu=smmuv3 “ when launching the QEMU.
>> I do use the option
> I used "-machine virt,iommu=smmuv3 “  option while creating the virt-dtb and 
> while launching the QEMU.
> I also observed the same error what you observed if I am not using the 
> "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought 
> this might be case for you also but anyways you have use the options it might 
> be other issue.

Hm, probably that was on my side as now I can see:

(XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
(XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration 
(false)
(XEN) SMMUv3: /smmuv3@9050000: no translation support!
(XEN) I/O virtualisation disabled
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Couldn't configure correctly all the IOMMUs.
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

So, sorry for the noise, I might have misconfigured something it seems

When you say "Xen is booting" do you mean you see the same panic?

Thank you,

Oleksandr

>
>>> Please let me know if it also works for you.
>> Well, I should have reported that earlier that I do not use the staging Xen 
>> at the moment,
>>
>> it is 4.14.0. So, can this be a problem with that Xen version?
> I don’t think so this is the problem with the XEN version.
>> Anyways, if it works with the staging then everything looks ok
>>
>> Thank you,
>>
>> Oleksandr
>>
>>>>> Regards
>>>>> Bertrand
>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Oleksandr
> Regards,
> Rahul