Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel <tamas.k.lengyel@xxxxxxxxx>]

On Mon, Feb 1, 2021 at 9:10 PM Roman Shaposhnik <roman@xxxxxxxxxx> wrote:
>
> On Mon, Feb 1, 2021 at 5:40 PM Stefano Stabellini
> <sstabellini@xxxxxxxxxx> wrote:
> >
> > On Mon, 1 Feb 2021, Tamas K Lengyel wrote:
> > > On Mon, Feb 1, 2021 at 10:23 AM Tamas K Lengyel
> > > <tamas.k.lengyel@xxxxxxxxx> wrote:
> > > >
> > > > On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@xxxxxxx> 
> > > > wrote:
> > > > >
> > > > > On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > > > > > With rpi-4.19.y kernel and dtbs
> > > > > > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > > > > > previous error is not present. I get the boot log on the serial with
> > > > > > just console=hvc0 from dom0 but the kernel ends up in a panic down 
> > > > > > the
> > > > > > line:
> > > > >
> > > > > > This seems to have been caused by a monitor being attached to the 
> > > > > > HDMI
> > > > > > port, with HDMI unplugged dom0 boots OK.
> > > > >
> > > > > The balance of reports seem to suggest 5.10 is the way to go if you 
> > > > > want
> > > > > graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety 
> > > > > on
> > > > > RP4.
> > > > >
> > > > >
> > > > > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > > > > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@xxxxxxx> 
> > > > > > wrote:
> > > > > > >
> > > > > > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > > > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell 
> > > > > > > > <ehem+undef@xxxxxxx> wrote:
> > > > > > > > > Presently the rpixen script is grabbing the RPF's 4.19 
> > > > > > > > > branch, dates
> > > > > > > > > point to that last being touched last year.  Their tree is at
> > > > > > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > > > > > >
> > > > > > > > I've moved the Linux branch up to 5.10 because there had been a 
> > > > > > > > fair
> > > > > > > > amount of work that went into fixing Xen on RPI4, which got 
> > > > > > > > merged
> > > > > > > > into 5.9 and I would like to be able to build upstream 
> > > > > > > > everything
> > > > > > > > without the custom patches coming with the rpixen script repo.
> > > > > > >
> > > > > > > Please keep track of where your kernel source is checked out at 
> > > > > > > since
> > > > > > > there was a desire to figure out what was going on with the 
> > > > > > > device-trees.
> > > > > > >
> > > > > > >
> > > > > > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" 
> > > > > > > in the
> > > > > > > kernel command-line should ensure you get output from the kernel 
> > > > > > > if it
> > > > > > > manages to start (yes, Linux does support having multiple 
> > > > > > > consoles at the
> > > > > > > same time).
> > > > > >
> > > > > > No output from dom0 received even with the added console options
> > > > > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > > > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still 
> > > > > > boots
> > > > > > with 4.19 next.
> > > > >
> > > > > So, their current HEAD.  This reads like you've got a problematic 
> > > > > kernel
> > > > > configuration.  What procedure are you following to generate the
> > > > > configuration you use?
> > > > >
> > > > > Using their upstream as a base and then adding the configuration 
> > > > > options
> > > > > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > > > > `make menuconfig`, `make zImage`).
> > > > >
> > > > > Notably the options:
> > > > > CONFIG_PARAVIRT
> > > > > CONFIG_XEN_DOM0
> > > > > CONFIG_XEN
> > > > > CONFIG_XEN_BLKDEV_BACKEND
> > > > > CONFIG_XEN_NETDEV_BACKEND
> > > > > CONFIG_HVC_XEN
> > > > > CONFIG_HVC_XEN_FRONTEND
> > > > >
> > > > > Should be set to "y".
> > > >
> > > > Yes, these configs are all set the same way for all Linux builds by the 
> > > > script:
> > > >         make O=.build-arm64 ARCH=arm64
> > > > CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> > > >
> > > > I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> > > > dom0. So far only 4.19 boots.
> > >
> > > rpi-5.4.y boots but ends up in yet another different kernel panic:
> >
> > That's an interesting error. However, I can tell you that I can boot
> > rpi-5.9.y just fine (without a monitor attached) with:
> >
> > cd linux
> > KERNEL=kernel7l
> > make bcm2711_defconfig
> >
> > As mentioned here:
> >
> > https://www.raspberrypi.org/documentation/linux/kernel/building.md
> >
> > and also taking the device tree from arch/arm64/boot/dts/broadcom/.
>
> FWIW: I see the same results with stock upstream 5.10.7 effectively
> following the steps you're doing.
>
> However, as I keep saying -- the combination of firmware and u-boot
> (in my case) is a very sensitive combination -- hence we're relying
> on a very particular set of bits for there in EVE and will refuse to work
> with anything else.
>
> It may be helpful to take that combination outside of EVE's context and
> try it out in your experiments Tamas.

Well, I'm giving up on this for now. I ran out of ideas to try and I
don't see any useful suggestions on how to debug this further. Looks
like it's super fragile and works only under specific conditions
that's not well documented - if documented at all.

Tamas