Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Roman Shaposhnik <roman@xxxxxxxxxx>]

On Fri, May 1, 2020 at 4:42 AM Corey Minyard <minyard@xxxxxxx> wrote:
>
> On Thu, Apr 30, 2020 at 07:20:05PM -0700, Roman Shaposhnik wrote:
> > Hi!
> >
> > I'm trying to run Xen on Raspberry Pi 4 with 5.6.1 stock,
> > upstream kernel. The kernel itself works perfectly well
> > on the board. When I try booting it as Dom0 under Xen,
> > it goes into a stacktrace (attached).
>
> Getting Xen working on the Pi4 requires a lot of moving parts, and they
> all have to be right.

Tell me about it! It is a pretty frustrating journey to align
everything just right.
On the other hand -- it seems worth to enable RPi as an ARM development
platform for Xen given how ubiquitous it is.

Hence me trying to combine pristine upstream kernel (5.6.1) with
pristine upstream
Xen to enable 100% upstream developer workflow on RPi.

> > Looking at what nice folks over at Dornerworks have previously
> > done to make RPi kernels boot as Dom0 I've come across these
> > 3 patches:
> >     
> > https://github.com/dornerworks/xen-rpi4-builder/tree/master/patches/linux
> >
> > The first patch seems irrelevant (unless I'm missing something
> > really basic here).
>
> It might be irrelevant for your configuration, assuming that Xen gets
> the right information from EFI.  I haven't tried EFI booting.

I'd doing a bit of belt-and-suspenders strategy really -- I'm actually
using UEFI u-boot implementation that pre-populates device trees
and then I'm also forcing an extra copy of it to be load explicitly
via the GRUB devicetree command (GRUB runs as a UEFI payload).

I also have access to the semi-official TianoCore RPi4 port that seems
to be working pretty well: https://github.com/pftf/RPi4/releases/tag/v1.5
for booting all sort of UEFI payloads on RPi4.

> > The 2nd patch applied with no issue (but
> > I don't think it is related) but the 3d patch failed to apply on
> > account of 5.6.1 kernel no longer having:
> >     dev->archdata.dev_dma_ops
> > E.g.
> >     
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/mm/dma-mapping.c?h=v5.6.1#n55
> >
> > I've tried to emulate the effect of the patch by simply introducing
> > a static variable that would signal that we already initialized
> > dev->dma_ops -- but that didn't help at all.
> >
> > I'm CCing Jeff Kubascik to see if the original authors of that Corey Minyard
> > to see if may be they have any suggestions on how this may be dealt
> > with.
> >
> > Any advice would be greatly appreciated!
>
> What's your Pi4 config.txt file look like? The GIC is not being handled
> correctly, and I'm guessing that configuration is wrong.  You can't use
> the stock config.txt file with Xen, you have to modify the configuration a
> bit.

Understood. I'm actually using a custom one:
    https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/config.txt

I could swear that I had a GIC setting in it -- but apparently no -- so
I added the following at the top of what you could see at the URL above:

total_mem=4096
enable_gic=1

> I think just adding:
>
> enable_gic=1
> total_mem=1024

Right -- but my board has 4G memory -- so I think what I did above should work.

> might get it working, or at least solve one problem.  It's required either
> way.  That might get rid of the GIC errors at the beginning.  I see a
> few of those, but not that many.
>
> My kernel command line is:
>
> coherent_pool=1M 8250.nr_uarts=1 cma=64M cma=256M  
> smsc95xx.macaddr=DC:A6:32:4F:3A:CD vc_mem.mem_base=0x3ec00000 
> vc_mem.mem_size=0x40000000  console=hvc0 clk_ignore_unused 
> root=/dev/mmcblk0p2 rootwait
>
> A lot of that configuration gets pulled from the initialization done by
> the GPU at startup which it put into the device tree.  I'm not sure what a
> lot of it means.  Some of it is added by Xen, too.

Just to be on the safe side -- I'm now using Device Tree from Dornerworks build.
The file is attached.

> I can verify the DMA patch is important.  I'm not sure how to apply it
> to a 5.6 kernel, though.

OK, after chatting with Stefano it definitely seems that there's some kind of
an issue with DMA. It just seems different to what the original issue with
4.19.x kernel used to be.

Here's the clue. When I disable Xen DMA operations altogether with
the following. I get the Linux kernel that is booting all the way to trying
mounting a root filesystem (which sadly can't be found since eMMC
requires DMA FWIU). I hope this will provide enough of a clue to
figure out what may be wrong with the 5.x kernel. I'm attaching the
boo log with the following patch applied:

diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
index d40e9e5fc52b..c25ead822de0 100644
--- a/arch/arm/xen/mm.c
+++ b/arch/arm/xen/mm.c
@@ -137,8 +137,7 @@ void xen_destroy_contiguous_region(phys_addr_t
pstart, unsigned int order)
 static int __init xen_mm_init(void)
 {
        struct gnttab_cache_flush cflush;
-       if (!xen_initial_domain())
-               return 0;
+       return 0;
        xen_swiotlb_init(1, false);

        cflush.op = 0;

Thanks,
Roman.

Attachment: log.txt
Description: Text document

Attachment: r.bin
Description: Binary data