[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU



Hi,

On 02/05/2020 03:16, Corey Minyard wrote:
On Fri, May 01, 2020 at 06:06:11PM -0700, Roman Shaposhnik wrote:
On Fri, May 1, 2020 at 4:42 AM Corey Minyard <minyard@xxxxxxx> wrote:

On Thu, Apr 30, 2020 at 07:20:05PM -0700, Roman Shaposhnik wrote:
Hi!

I'm trying to run Xen on Raspberry Pi 4 with 5.6.1 stock,
upstream kernel. The kernel itself works perfectly well
on the board. When I try booting it as Dom0 under Xen,
it goes into a stacktrace (attached).

Getting Xen working on the Pi4 requires a lot of moving parts, and they
all have to be right.

Tell me about it! It is a pretty frustrating journey to align
everything just right.
On the other hand -- it seems worth to enable RPi as an ARM development
platform for Xen given how ubiquitous it is.

Hence me trying to combine pristine upstream kernel (5.6.1) with
pristine upstream
Xen to enable 100% upstream developer workflow on RPi.

Looking at what nice folks over at Dornerworks have previously
done to make RPi kernels boot as Dom0 I've come across these
3 patches:
     https://github.com/dornerworks/xen-rpi4-builder/tree/master/patches/linux

The first patch seems irrelevant (unless I'm missing something
really basic here).

It might be irrelevant for your configuration, assuming that Xen gets
the right information from EFI.  I haven't tried EFI booting.

I'd doing a bit of belt-and-suspenders strategy really -- I'm actually
using UEFI u-boot implementation that pre-populates device trees
and then I'm also forcing an extra copy of it to be load explicitly
via the GRUB devicetree command (GRUB runs as a UEFI payload).

I also have access to the semi-official TianoCore RPi4 port that seems
to be working pretty well: https://github.com/pftf/RPi4/releases/tag/v1.5
for booting all sort of UEFI payloads on RPi4.

The 2nd patch applied with no issue (but
I don't think it is related) but the 3d patch failed to apply on
account of 5.6.1 kernel no longer having:
     dev->archdata.dev_dma_ops
E.g.
     
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/mm/dma-mapping.c?h=v5.6.1#n55

I've tried to emulate the effect of the patch by simply introducing
a static variable that would signal that we already initialized
dev->dma_ops -- but that didn't help at all.

I'm CCing Jeff Kubascik to see if the original authors of that Corey Minyard
to see if may be they have any suggestions on how this may be dealt
with.

Any advice would be greatly appreciated!

What's your Pi4 config.txt file look like? The GIC is not being handled
correctly, and I'm guessing that configuration is wrong.  You can't use
the stock config.txt file with Xen, you have to modify the configuration a
bit.

Understood. I'm actually using a custom one:
     https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/config.txt

I could swear that I had a GIC setting in it -- but apparently no -- so
I added the following at the top of what you could see at the URL above:

total_mem=4096
enable_gic=1

I think just adding:

enable_gic=1
total_mem=1024

Right -- but my board has 4G memory -- so I think what I did above should work.

Nope.  If you say 4096M of RAM, your issue is almost certainly DMA, but
it's not (just) the Linux code.  On the Raspberry Pi 4, several devices
cannot DMA to above 1024M of RAM, but Xen does not handle this.  The
1024M of RAM is a limitation you will have to live with until Xen has a
fix.

IIUC, dom0 would need to have some memory below 1GB for this to work, am I correct?

If so could you try the following patch?

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 430708753642..002f49dba74b 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -282,7 +282,7 @@ static void __init allocate_memory_11(struct domain *d,
      */
     while ( order >= min_low_order )
     {
-        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
+        for ( bits = order ; bits <= (lowmem ? 30 : PADDR_BITS); bits++ )
         {
             pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
             if ( pg != NULL )
@@ -313,7 +313,7 @@ static void __init allocate_memory_11(struct domain *d,
     order = get_allocation_size(kinfo->unassigned_mem);
     while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS )
     {
-        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(32) : 0);
+        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(30) : 0);
         if ( !pg )
         {
             order --;


Cheers,

--
Julien Grall



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.