AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer

[Leo Krueger <leo.krueger@xxxxxxxx>]

Hi,

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xxxxxxx>
> Gesendet: Montag, 23. November 2020 19:27
> An: Leo Krueger <leo.krueger@xxxxxxxx>; Stefano Stabellini
> <stefano.stabellini@xxxxxxxxxx>
> Cc: Peng Fan <peng.fan@xxxxxxx>; brucea@xxxxxxxxxx; Cornelia Bruelhart
> <cornelia.bruelhart@xxxxxxxx>; oleksandr_andrushchenko@xxxxxxxx; xen-
> devel@xxxxxxxxxxxxxxxxxxxx; Bertrand.Marquis@xxxxxxx
> Betreff: Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
> 
> 
> 
> On 22/11/2020 22:55, Leo Krueger wrote:
> > Hi Julien,
> 
> Hi Leo,
> 
> >
> > finally I could try out what you suggested, please find my answers inline.
> 
> Thank you for sending the logs!
> 
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Julien Grall <julien@xxxxxxx>
> >> Gesendet: Mittwoch, 18. November 2020 13:24
> >> An: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Leo Krueger
> >> <leo.krueger@xxxxxxxx>
> >> Cc: Peng Fan <peng.fan@xxxxxxx>; brucea@xxxxxxxxxx; Cornelia
> >> Bruelhart <cornelia.bruelhart@xxxxxxxx>;
> >> oleksandr_andrushchenko@xxxxxxxx; xen- devel@xxxxxxxxxxxxxxxxxxxx;
> >> Bertrand.Marquis@xxxxxxx
> >> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> >>
> >> Hi,
> >>
> >> On 17/11/2020 23:53, Stefano Stabellini wrote:
> >>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> >>> recent experience with GICv3 ITS than me and might be able to help.
> >>> I am attaching the device tree Leo sent a few days ago for reference.
> >>>
> >>>
> >>> Typically when you can set the ethernet link up and no packets are
> >>> exchanged it is because of a missing interrupt. In this case a
> >>> missing MSI.
> >>>
> >>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> >>> recently. It is expected to work correctly with MSIs in Dom0, right?
> >>
> >> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to
> >> boot Dom0. I haven't seen any failure on recent Xen. We are testing
> >> 4.11 and onwards on Thunder-X.
> >>
> >> However, it may be possible that some more work is necessary for
> >> other hardware (e.g. workaround, missing code...). See more below.
> >>
> >>>
> >>>
> >>>
> >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> >>>> Hi,
> >>>>
> >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >>>> before...) but then had to add the following node to my device tree
> >>>>
> >>>>  gic_lpi_base: syscon@0x80000000 {
> >>>>          compatible = "gic-lpi-base";
> >>
> >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> >> could you clarify which flavor/version of Linux you are using?
> >
> > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> 
> Do you have a link to the Linux tree? Is there any additional patches on top 
> of
> vanilla?

Linux tree is found here: 
https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
(up to the latest commit in that branch)

> 
> > While searching around the Internet for any solution, I came across [0]
> which contained the gic-lpi-base node.
> > So I just tried adding it (quite desperate I know) and voila, it at least
> brought me one step further (XEN exposing the ITS)...
> 
> I am slightly confused to how this would help. Xen and, AFAICT, Linux don't
> understand gic-lpi-base. Do you have modification in your Linux to use it?

I have no idea, to be honest. Maybe it is about the memory being reserved due 
to that node or something like that?

> 
> Looking at the DT changes in [0], it looks like the node is not a child of 
> gic@.
> So I think Xen will map the region to Dom0.
> 
> There are two things that I can notice:
>    1) This region is RAM, but I can't find any reserve node. Is there any 
> specific
> code in Linux to reserve it?
>    2) The implementation in U-boot seems to suggest that the firmware will
> configure the LPIs and then enable it. If that's the case, then Xen needs to
> re-use the table in the DT rather than allocating a new one.
> However, I would have expected an error message in the log:
> 
>     "GICv3: CPUx: Cannot initialize LPIs"
> 
> At least Xen should not expose gic-lpi-base to the kernel, but I will wait on
> more details about the Linux kernel used before commenting more.
> 
> I would also be interested to know more details about the failure when gic-
> lpi-base is not added in your DT. In particular, I am interested to understand
> why Xen would not expose the ITS as we don't parse that node.

How can I supply you with more information in regard to that? Without that 
node, ITS was not exposed at all.

> 
> [...]
> 
> > For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I 
> > know,
> quite ugly in parts).
> 
> No worries, debug patches are not meant to be nice to read ;).
> 
> > Find attached the boot log and an output of "xl dmesg" which is truncated
> due to the large amount of messages.
> >
> > When enabling the network interface (gbe0), the following output is
> visible:
> >
> > root@kontron-sal28:~# ip link set up dev gbe0
> > (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c
> > 0000000000000001 0000000000000000 0000000000000000
> > (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005
> > 0000000000000000 0000000000000000 0000000000000000
> 
> 0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt by
> writing in the property table (access are not trapped to Xen) and then
> requested to invalidate the cache state.
> 
> > [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver
> [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> > [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> > [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> > root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is
> Down
> > [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow
> control off
> > [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes
> ready
> >
> > Does that tell you anything?
> 
> It is at least a good sign because it means Linux is able to initialize/talk 
> to the
> vITS.
> 
> I would lean towards one (or multiple) issue with pITS and/or the device-tree
> exposed to Linux. I am not entirely what exactly... I think having more 
> details
> about the Linux setup would be helpful.

Ok let me know what you need from my side. I was considering the following 
things to try out next:

- a more recent u-boot version as this might fix problems with the msi-map (at 
least that is what I think, I am not an expert here)
- a different device tree (a more recent one, ...)
- ...

> 
> I will reply on Rahul's e-mail separately.
> 
> Cheers,

Best wishes!

> 
> --
> Julien Grall