Re: [Xen-devel] [GSoC] Xen on ARM: create multiple guests from device tree

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 23 Mar 2017, Luca Miccio wrote:
> > Il giorno 23 mar 2017, alle ore 01:27, Stefano Stabellini 
> > <sstabellini@xxxxxxxxxx> ha scritto:
> > 
> > On Thu, 23 Mar 2017, Luca Miccio wrote:
> >> Hi Stefano and Julien,
> >> 
> >>> Il giorno 22 mar 2017, alle ore 22:38, Stefano Stabellini 
> >>> <sstabellini@xxxxxxxxxx> ha scritto:
> >>> 
> >>> Hi Luca,
> >>> 
> >>> please don't use HTML emails.
> >>> 
> >>> 
> >> Sorry for that.
> >> 
> >>> On Wed, 22 Mar 2017, Julien Grall wrote:
> >>>> On 22/03/2017 19:45, Luca Miccio wrote:
> >>>>> Hi Stefano,
> >>>> 
> >>>> Hello Luca,
> >>>> 
> >>>>> aarch64-linux-gnu-gcc  -DCNTFRQ=0x01800000-DUART_BASE=0x1c090000
> >>>>> -DSYSREGS_BASE=0x1c010000 -DGIC_DIST_BASE=0x2c001000
> >>>>> -DGIC_CPU_BASE=0x2c002000 -c -o boot.xen.o boot.S -DXEN
> >>>>> aarch64-linux-gnu-gcc  -DPHYS_OFFSET=0x80000000 -DMBOX_OFFSET=0xfff8
> >>>>> -DBOOT=boot.xen.o -DXEN_OFFSET=0xA00000 -DKERNEL_OFFSET=0x80000
> >>>>> -DFDT_OFFSET=0x08000000 -DFS_OFFSET=0x10000000 -DXEN=Xen -DKERNEL=Image
> >>>>> -DFILESYSTEM=filesystem.cpio.gz -E -P -C -o model.xen.lds model.lds.S
> >>>>> aarch64-linux-gnu-gcc  -DCNTFRQ=0x01800000-DUART_BASE=0x1c090000
> >>>>> -DSYSREGS_BASE=0x1c010000 -DGIC_DIST_BASE=0x2c001000
> >>>>> -DGIC_CPU_BASE=0x2c002000 -c -o boot.o boot.S
> >>>>> aarch64-linux-gnu-gcc  -DPHYS_OFFSET=0x80000000 -DMBOX_OFFSET=0xfff8
> >>>>> -DBOOT=boot.o -DKERNEL_OFFSET=0x80000 -DFDT_OFFSET=0x08000000
> >>>>> -DFS_OFFSET=0x10000000 -DKERNEL=Image -DFILESYSTEM=filesystem.cpio.gz -E
> >>>>> -P -C -o model.lds model.lds.S
> >>>>> aarch64-linux-gnu-ld -o xen-system.axf --script=model.xen.lds
> >>>>> aarch64-linux-gnu-ld: section .xen LMA
> >>>>> [0000000080a00000,0000000080ac814f] overlaps section .kernel LMA
> >>>>> [0000000080080000,0000000080f009ff]
> >>>>> Makefile:78: set di istruzioni per l'obiettivo "xen-system.axf" non 
> >>>>> riuscito
> >>>>> make: *** [xen-system.axf] Errore 1\
> >>>>> 
> >>>>> Clearly there is a problem with the offset of xen section but i can't
> >>>>> figure out how to solve it.
> >>>> 
> >>>> I would recommend to use the latest version of bootwrapper which is now
> >>>> including support for Xen. I haven't yet updated the wiki with the runes
> >>>> but here a quick summary of the step I am using:
> >>>> 
> >>>> * Building bootwrapper
> >>>> 
> >>>> 42sh> git clone 
> >>>> git://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git
> >>>> 42sh> autoreconf -i
> >>>> 42sh> ./configure --host=aarch64-linux-gnu 
> >>>> --with-kernel-dir=<kernel-dir> --with-dtb=<dtb> \
> >>>> --with-cmdline="console=hvc0 earlycon=pl011,0x1c090000 root=/dev/vda rw" 
> >>>> --enable-psci \
> >>>> --with-xen-cmdline="dtuart=serial0 console=dtuart no-bootscrub 
> >>>> dom0_mem=512M" \
> >>>> --with-xen=<xen-bin> --with-cpu-ids=0,1,2,3
> >>>> 42sh> make
> >>>> 
> >>>> Where:
> >>>> * <kernel-dir> is the path to the kernel repository
> >>>> * <dtb> is the path to the device-tree. I am using the one shipped with 
> >>>> Linux
> >>>>   in arch/arm64/boot/dts/arm/foundation-armv8.dtbs
> >>>>   It will be built if you did make all in Linux
> >>>> * <xen-bin> is the path to Xen binary (xen/xen in the repo)
> >>>> 
> >>>> * Command line for the foundation model
> >>>> 42sh> Foundation_Platform --image=<image> --block-device=<rootfs> 
> >>>> --cores=4
> >>>> 
> >>>> Where:
> >>>> * <image> is the path to xen-system.axf produced by bootwrapper
> >>>> * <rootfs> is the patch to your rootfs
> >>>> 
> >>>> Let me know if you need more details.
> >>> 
> >>> Thanks Julien, I successfully booted Xen and Linux following these
> >>> steps. I quickly updated the wiki.
> >>> 
> >>> Luca, I took me a while to figure out that for root=/dev/vda to work
> >>> properly, Linux requires not just all the filesystem and virtio options
> >>> enabled in the kernel config, but also CONFIG_VIRTIO_MMIO=y. With that,
> >>> you should be able to boot Xen, Linux and mount the root filesystem. If
> >>> you download one of the Ubuntu arm64 cloud images, the root device will
> >>> be /dev/vda1.
> >> 
> >> First of all, thank you for your help.
> >> 
> >> I think that i keep on doing somenthing wrong because now the bootwrapper 
> >> produces
> >> with no errors the xen-system.axf  with the configuration that Julien 
> >> provided. 
> >> But when i launch the emulator it simply stops at:
> >> (XEN) Bringing up CPU1
> >> 
> >> To help you understand, this is what i have done:
> >> - First i compiled the xen source with the following command:
> >> $ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 
> >> CONFIG_EARLY_PRINT=fastmodel
> >> ( I am using the staging branch)
> >> 
> >> - Get the linux source with git: 
> >> $ git clone 
> >> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >> 
> >> -Checkout on the v4.4 branch 
> >> 
> >> - Follow the steps described in the blog and, as you said, i set in my 
> >> .config file:
> >> CONFIG_VIRTIO=y
> >> CONFIG_VIRTIO_MMIO=y
> >> CONFIG_VIRTIO_BLK=y
> >> I also checked all the filesystem and virtio options. 
> >> 
> >> - Compile the linux kernel with the toolchain:
> >> $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- all (to get also the 
> >> .dtb file)
> >> 
> >> - Configured the bootwrapper with the command that Julian wrote and then 
> >> launched the 
> >> foundation model with the simple command:
> >> $ Foundation_Platform --image=/path/to/the/xen-system.axf 
> >> --block-device=xenial-server-cloudimg-arm64.img --cores=4
> > 
> > Try removing --cores=4, does it work?
> > 
> I have already tried this option with no results

The problem is that no matter the --cores=4 option, the device tree
(foundation-v8.dtb) has 4 cpu cores by default. It also says that the
"enable-method" is "spin-table". At the same time Julien wrote to pass
--enable-psci to the bootwrapper, which is a different way to start
secondary cpus. I am guessing that is the problem. I suggest to change
boot method to psci, to do that you need to modify the dts (device tree
source), see the attached patch for Linux. Then you can rebuild the dtb
(device tree binary) with:

make dtbs

Finally rebuild again the bootwrapper and run the model with --cores 4.

For me, that works. Great way to get familiar with device tree ;-)


> >> The image was downloaded at : 
> >> https://cloud-images.ubuntu.com/releases/16.04/release/
> >> 
> >> But, as I said, the emulator blocks at boot. I tried different rootfs 
> >> images and i have configured the root device
> >> properly to /dev/vda1.
> >> 
> >> Maybe my kernel configuration is wrong?

Attachment: 1
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel