Re: Question about xen and Rasp 4B

[Jukka Kaartinen <jukka.kaartinen@xxxxxxxxxx>]

On 3.2.2021 22.55, Stefano Stabellini wrote:
> On Wed, 3 Feb 2021, Jukka Kaartinen wrote:
> > On 3.2.2021 2.18, Stefano Stabellini wrote:
> > > On Tue, 2 Feb 2021, Jukka Kaartinen wrote:
> > > > > > Good catch.
> > > > > > GPU works now and I can start X! Thanks! I was also able to create
> > > > > > domU
> > > > > > that runs Raspian OS.
> > > > > This is very interesting that you were able to achieve that - congrats!
> > > > >
> > > > > Now, sorry to be a bit dense -- but since this thread went into all
> > > > > sorts of interesting
> > > > > directions all at once -- I just have a very particular question: what
> > > > > is
> > > > > exact
> > > > > combination of versions of Xen, Linux kernel and a set of patches that
> > > > > went
> > > > > on top that allowed you to do that? I'd love to obviously see it
> > > > > productized in Xen
> > > > > upstream, but for now -- I'd love to make available to Project EVE/Xen
> > > > > community
> > > > > since there seems to be a few folks interested in EVE/Xen combo being
> > > > > able
> > > > > to
> > > > > do that.
> > > > I have tried Xen Release 4.14.0, 4.14.1 and master (from week 4, 2021).
> > > >
> > > > Kernel rpi-5.9.y and rpi-5.10.y branches from
> > > > https://github.com/raspberrypi/linux
> > > >
> > > > and
> > > >
> > > > U-boot (master).
> > > >
> > > > For the GPU to work it was enough to disable swiotlb from the kernel(s) as
> > > > suggested in this thread.
> > > How are you configuring and installing the kernel?
> > >
> > > make bcm2711_defconfig
> > > make Image.gz
> > > make modules_install
> > >
> > > ?
> > >
> > > The device tree is the one from the rpi-5.9.y build? How are you loading
> > > the kernel and device tree with uboot? Do you have any interesting
> > > changes to config.txt?
> > >
> > > I am asking because I cannot get to the point of reproducing what you
> > > are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
> > > get any graphics output on my screen. (The serial works.) I am using the
> > > default Ubuntu Desktop rpi-install target as rootfs and uboot master.
> > This is what I do:
> >
> > make bcm2711_defconfig
> > cat "xen_additions" >> .config
> > make Image  modules dtbs
> >
> > make INSTALL_MOD_PATH=rootfs modules_install
> > depmod -a
> >
> > cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb boot/
> > cp arch/arm64/boot/dts/overlays/*.dtbo boot/overlays/
> Thanks for the detailed instructions. This helps a lot. I saw below in
> boot2.source that you are using ${fdt_addr} as DTB source (instead of
> loading one), which means you are using the DTB as provided by U-Boot at
> runtime, instead of loading your own file.
>
> With these two copies, I take you meant to update the first partition on
> the SD card, the one where config.txt lives, right? So that Xen is
> getting the DTB and overlays from the rpi-5.9.y kernel tree but passed
> down by the RPi loader and U-Boot?
>
> I think the DTB must be the issue as I wasn't applying any overlays
> before. I ran a test to use the DTB and overlay from U-Boot but maybe I
> haven't updated them properly because I still don't see any output.
I'm using ${fdt_addr} has the changes that FW will do according to your 
congig.txt for example the overlay is applied. I tried to load and apply 
the overlay in the u-boot but there were strange errors so I decided to 
go the easy way.
I the overlay (vc4-fkms-v3d) is not applied GPU (v3d) driver is not probed.

I always see output from u-boot from the serial port. If not then there 
has been something wrong with the device-tree. And it has been because 
u-boot was not able to read from the SSD.
> > config.txt:
> >
> > [pi4]
> > max_framebuffers=2
> > enable_uart=1
> > arm_freq=1500
> > force_turbo=1
> >
> > [all]
> > arm_64bit=1
> > kernel=u-boot.bin
> >
> > start_file=start4.elf
> > fixup_file=fixup4.dat
> >
> > # Enable the audio output, I2C and SPI interfaces on the GPIO header
> > dtparam=audio=on
> > dtparam=i2c_arm=on
> > dtparam=spi=on
> >
> > # Enable the FKMS ("Fake" KMS) graphics overlay, enable the camera firmware
> > # and allocate 128Mb to the GPU memory
> > dtoverlay=vc4-fkms-v3d,cma-64
> > gpu_mem=128
> >
> > # Comment out the following line if the edges of the desktop appear outside
> > # the edges of your display
> > disable_overscan=1
> >
> >
> > boot.source:
> > setenv serverip 10.42.0.1
> > setenv ipaddr 10.42.0.231
> > tftpb 0xC00000 boot2.scr
> > source 0xC00000
> >
> > boot2.source:
> > tftpb 0xE00000 xen
> > tftpb 0x1000000 Image
> > setenv lin_size $filesize
> >
> > fdt addr ${fdt_addr}
> > fdt resize 1024
> >
> > fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 sync_console
> > dom0_mem=1024M dom0_max_vcpus=1 bootscrub=0 vwfi=native sched=credit2"
> >
> > fdt mknod /chosen dom0
> >
> > # These will break the default framebuffer@3e2fe000 that
> > # is the same chosen -node.
> > #fdt set /chosen/dom0 \#address-cells <0x2>
> > #fdt set /chosen/dom0 \#size-cells <0x2>
> >
> > fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
> > fdt set /chosen/dom0 reg <0x1000000 0x${lin_size}>
> >
> > fdt set /chosen xen,dom0-bootargs "dwc_otg.lpm_enable=0 console=hvc0
> > earlycon=xen earlyprintk=xen root=/dev/sda4 elevator=deadline rootwait fixrtc
> > quiet splash"
> >
> > setenv fdt_high 0xffffffffffffffff
> >
> > fdt print /chosen
> >
> > #xen
> > booti 0xE00000 - ${fdt_addr}