Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 13 May 2020, Julien Grall wrote:
> On 13/05/2020 16:11, Stefano Stabellini wrote:
> > On Wed, 13 May 2020, Julien Grall wrote:
> > > Hi,
> > > 
> > > On 13/05/2020 01:33, Stefano Stabellini wrote:
> > > > I worked with Roman to do several more tests and here is an update on
> > > > the situation. We don't know why my patch didn't work when Boris' patch
> > > > [1] worked.  Both of them should have worked the same way.
> > > > 
> > > > Anyway, we continued with Boris patch to debug the new mmc error which
> > > > looks like this:
> > > > 
> > > > [    3.084464] mmc0: unrecognised SCR structure version 15
> > > > [    3.089176] mmc0: error -22 whilst initialising SD card
> > > > 
> > > > I asked to add a lot of trancing, see attached diff.
> > > 
> > > Please avoid attachment on mailing list and use pastebin for diff.
> > > 
> > > > We discovered a bug
> > > > in xen_swiotlb_init: if io_tlb_start != 0 at the beginning of
> > > > xen_swiotlb_init, start_dma_addr is not set correctly. This oneline
> > > > patch fixes it:
> > > > 
> > > > diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> > > > index 0a40ac332a4c..b75c43356eba 100644
> > > > --- a/drivers/xen/swiotlb-xen.c
> > > > +++ b/drivers/xen/swiotlb-xen.c
> > > > @@ -191,6 +191,7 @@ int __ref xen_swiotlb_init(int verbose, bool early)
> > > >            * IO TLB memory already allocated. Just use it.
> > > >            */
> > > >           if (io_tlb_start != 0) {
> > > > +               start_dma_addr = io_tlb_start;
> > > >                   xen_io_tlb_start = phys_to_virt(io_tlb_start);
> > > >                   goto end;
> > > >           }
> > > > 
> > > > Unfortunately it doesn't solve the mmc0 error.
> > > > 
> > > > 
> > > > As you might notice from the logs, none of the other interesting printks
> > > > printed anything, which seems to mean that the memory allocated by
> > > > xen_swiotlb_alloc_coherent and mapped by xen_swiotlb_map_page should be
> > > > just fine.
> > > > 
> > > > I am starting to be out of ideas. Do you guys have any suggestions on
> > > > what could be the issue or what could be done to discover it?
> > > 
> > > Sorry if my suggestions are going to be obvious, but I can't confirm
> > > whether
> > > this was already done:
> > >      1) Does the kernel boot on baremetal (i.e without Xen)? This should
> > > help
> > > to confirm whether the bug is Xen is related.
> > 
> > Yes it boots
> > 
> > >      2) Swiotlb should not be necessary for basic dom0 boot on Arm. Did
> > > you try
> > > to disable it? This should help to confirm whether swiotlb is the problem
> > > or
> > > not.
> > 
> > It boots disabling swiotlb-xen
> 
> Thank you for the answer! swiotlb-xen should basically be a NOP for dom0. So
> this suggests swiotlb is doing some transformation on the DMA address.

Yes, but couldn't spot where yet


> I have an idea what may have gone wrong. AFAICT, xen-swiotlb seems to assume
> the DMA address space and CPU address space is the same. Is RPI using the same
> address space?

I thought about it too but we didn't add an explicit check for this yet.
My understanding is that it is not supposed to happen on arm64 boards
but at this point it is worth checking it for sure. Thanks for the
suggestion.