Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM

[Roman Shaposhnik <roman@xxxxxxxxxx>]

On Tue, Dec 17, 2019 at 6:56 PM Roman Shaposhnik <roman@xxxxxxxxxx> wrote:
>
> On Tue, Dec 17, 2019 at 5:51 PM Stefano Stabellini
> <sstabellini@xxxxxxxxxx> wrote:
> >
> > On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > > On Tue, Dec 17, 2019 at 11:26 AM Stefano Stabellini
> > > <sstabellini@xxxxxxxxxx> wrote:
> > > >
> > > > On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > > > > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini
> > > > > <sstabellini@xxxxxxxxxx> wrote:
> > > > > >
> > > > > > On Tue, 17 Dec 2019, Julien Grall wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > On 17/12/2019 04:39, Roman Shaposhnik wrote:
> > > > > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
> > > > > > > > <sstabellini@xxxxxxxxxx> wrote:
> > > > > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > > > > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which 
> > > > > > > > > is 990M.
> > > > > > > > > If so, I wonder how you could boot succesfully with 
> > > > > > > > > dom0_mem=1024M even
> > > > > > > > > on Xen 4.12... :-?
> > > > > > > >
> > > > > > > > That is a very interesting observation indeed! I actually don't
> > > > > > > > remember where that device tree came from, but I think it was 
> > > > > > > > from one
> > > > > > > > of the Linaro sites.
> > > > > > >
> > > > > > > This is mostly likely because of:
> > > > > > >
> > > > > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e
> > > > > > > Author: Julien Grall <julien.grall@xxxxxxx>
> > > > > > > Date:   Wed Aug 21 22:42:31 2019 +0100
> > > > > > >
> > > > > > >     xen/arm: domain_build: Don't continue if unable to allocate 
> > > > > > > all dom0 banks
> > > > > > >
> > > > > > >     Xen will only print a warning if there are memory unallocated 
> > > > > > > when using
> > > > > > >     1:1 mapping (only used by dom0). This also includes the case 
> > > > > > > where no
> > > > > > >     memory has been allocated.
> > > > > > >
> > > > > > >     It will bring to all sort of issues that can be hard to 
> > > > > > > diagnostic for
> > > > > > >     users (the warning can be difficult to spot or disregard).
> > > > > > >
> > > > > > >     If the users request 1GB of memory, then most likely they 
> > > > > > > want the exact
> > > > > > >     amount and not 512MB. So panic if all the memory has not been 
> > > > > > > allocated.
> > > > > > >
> > > > > > >     After this change, the behavior is the same as for non-1:1 
> > > > > > > memory
> > > > > > >     allocation (used by domU).
> > > > > > >
> > > > > > >     At the same time, reflow the message to have the format on a 
> > > > > > > single
> > > > > > >     line.
> > > > > > >
> > > > > > >     Signed-off-by: Julien Grall <julien.grall@xxxxxxx>
> > > > > > >     Acked-by: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> > > > > >
> > > > > > Ah! Roman, could you please post the full boot log of a successful 
> > > > > > 4.12
> > > > > > boot?
> > > > > >
> > > > > > If it has a "Failed to allocate requested dom0 memory" message, 
> > > > > > then we
> > > > > > know what the issue is.
> > > > >
> > > > > Aha! Our messages seems to have crossed ;-) Full log is attached and
> > > > > yes -- that's
> > > > > the problem indeed.
> > > > >
> > > > > So at least that mystery is solved. But I'm still not able to get to a
> > > > > full 1G of memory
> > > > > even with your update to the device tree file. Any chance you can 
> > > > > send me the
> > > > > device tree file that works for you?
> > > >
> > > > I didn't try on real hardware, I only tried on QEMU with a similar
> > > > configuration. I went back and check the HiKey device tree I used and it
> > > > is the same as yours (including the ramoops reserved-memory error).
> > > >
> > > > Apparently there are 1G and 2G variants of the HiKey, obviously both
> > > > yours and my device tree are for the 1G variant. I try to dig through
> > > > the docs but couldn't find the details of the 2G variant. I cannot find
> > > > anywhere the memory range for the top 1G of memory not even on the
> > > > LeMaker docs! :-/
> > >
> > > Yup. That's exactly the issue on my end as well - can't seem to find an
> > > authoritative source for that devicetree.
> > >
> > > I did find this, though:
> > >      https://releases.linaro.org/96boards/hikey/linaro/debian/15.11/
> > > which looks like it has the latest (at least file timestamp-wise) 
> > > devicetree.
> > >
> > > If you look at the memory and reserved memory nodes there, they
> > > are actually much simpler than what we've got:
> > >
> > >         memory {
> > >                 device_type = "memory";
> > >                 reg = <0x0 0x0 0x0 0x40000000>;
> > >         };
> >
> > Which is still 1G, but it is surprisingly simpler.
> >
> >
> > >         reserved-memory {
> > >                 #address-cells = <0x2>;
> > >                 #size-cells = <0x2>;
> > >                 ranges;
> > >
> > >                 mcu-buf@05e00000 {
> > >                         no-map;
> > >                         reg = <0x0 0x5e00000 0x0 0x100000 0x0
> > > 0x740f000 0x0 0x1000>;
> > >                 };
> > >
> > >                 mbox-buf@06dff000 {
> > >                         no-map;
> > >                         reg = <0x0 0x6dff000 0x0 0x1000>;
> > >                 };
> > >         };
> > >
> > > So -- just on a whim -- I changed it to:
> > >     reg = <0x0 0x0 0x0 0x80000000>;
> >
> > I would have tried that too :-)
> >
> >
> > > Interestingly enough, Xen booted, and complained about only 192MB
> > > unallocated this time.
> > > So, I dropped the size of Dom0 to 640M and I got it boot and here's
> > > what I'm seeing as
> > > an output of xl info:
> > >    total_memory           : 1120
> > >    free_memory            : 390
> > > It still nowhere close to 2G.
> > >
> > > Then I booted the Linux kernel without Xen and it correctly identified
> > > all 2G worth of RAM, and in fact,
> >
> > Good! We can work with that.
>
> So that is, in fact, my first question -- why is Xen not showing
> available memory in xl info?
>
> > > when I converted /sys/firmware/devicetree/base back into dts, here's
> > > what I've got:
> > >
> > >         memory {
> > >                 device_type = "memory";
> > >                 reg = <0x0 0x0 0x0 0x5e00000 0x0 0x5f00000 0x0 0x1000
> > > 0x0 0x5f02000 0x0 0xefd000 0x0 0x6e00000 0x0 0x60f000 0x0 0x7410000
> > > 0x0 0x1aaf0000 0x0 0x21f00000 0x0 0x100000 0x0 0x22000000 0x0
> > > 0x1c000000>;
> > >         };
> > >
> > >         reserved-memory {
> > >                 ranges;
> > >                 #size-cells = <0x2>;
> > >                 #address-cells = <0x2>;
> > >
> > >                 ramoops@21f00000 {
> > >                         ftrace-size = <0x20000>;
> > >                         console-size = <0x20000>;
> > >                         reg = <0x0 0x21f00000 0x0 0x100000>;
> > >                         record-size = <0x20000>;
> > >                         compatible = "ramoops";
> > >                 };
> > >
> > >                 linux,cma {
> > >                         linux,cma-default;
> > >                         reusable;
> > >                         size = <0x0 0x8000000>;
> > >                         compatible = "shared-dma-pool";
> > >                 };
> > >         };
> > >
> > > If you look at the REG -- it does now add up to 2Gb,
> >
> > I am a bit confused by this. I did the calculation twice and it is still
> > only 990MB. In fact, what you pasted here really looks like the old
> > device tree. Is it possible that you run the test with the old device
> > tree?
>
> There's something weird going on when it comes to Linux on this
> box. Basically, it seems that regardless of what devicetree I pass,
> Linux kernel seems to successfully discover all of 2G of memory.
>
> For example, I'm attaching a Linux boot log that clearly shows:
>    Memory: 1877336K/2062392K available
> Even though I double checked that devicetree is only advertising 1G.
>
> But see below for more on this:
>
> > > but booting Xen
> > > with it has exactly theo
> > > same effect as booting it with: reg = <0x0 0x0 0x0 0x80000000>;
> > >
> > > I am attaching a full log, and I see the following in the logs:
> > >
> > > (XEN) Allocating 1:1 mappings totalling 720MB for dom0:
> > > (XEN) BANK[0] 0x00000008000000-0x0000001c000000 (320MB)
> > > (XEN) BANK[1] 0x00000040000000-0x00000058000000 (384MB)
> > > (XEN) BANK[2] 0x0000007b000000-0x0000007c000000 (16MB)
> > >
> > > Which sort of makes sense, I guess -- but I still don't understand
> > > where all these ranges
> > > are coming from and how come Xen doesn't see the full 2Gb even with 
> > > various
> > > devicetrees I tried.
> > >
> > > Any ideas here would be greatly apprecaited!
> >
> > I think you might have run the test with the old device tree by mistake?
> > If you are sure that Linux can boot OK with memory as:
> >
> >   reg = <0x0 0x0 0x0 0x80000000>;
> >
> > and correctly sees 2GB, then it should work with Xen too.
>
> Well, that's the issue -- it seems that Linux somehow doesn't depend *at all*
> on what I put in devicetrees -- it always detects full 2G.
>
> > In fact, looking at the logs you pasted, the choice of memory for dom0:
> >
> >   0x40000000-0x58000000
> >   0x7b000000-0x7c000000
> >
> > means that Xen was succesfully able to see the RAM above 0x40000000!
> > So, it looked like it already worked some extent!
>
> Exactly! That's the other surprising bit -- I noticed that too -- its not like
> Xen doesn't see any of the memory above 1G -- it just doesn't see enough of 
> it.
>
> So the question is -- what is Linux doing that Xen doesn't?

By the way, speaking of running Xen under ARM/qemu -- here's an interesting
observation: when I run qemu-system-aarch64 with -m 4096 option it seems
that, again, Linux kernel is perfectly content with having access to 4G of RAM,
while Xen only sees about 2G.

This may actually have something to do with UEFI I guess.

Would be great to get to the bottom of this.

Thanks,
Roman.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel