Re: domU reboot claim failed

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 10/09/2025 7:58 pm, Jason Andryuk wrote:
> Hi,
>
> We're running Android as a guest and it's running the Compatibility
> Test Suite.  During the CTS, the Android domU is rebooted multiple times.
>
> In the middle of the CTS, we've seen reboot fail.  xl -vvv shows:
> domainbuilder: detail: Could not allocate memory for HVM guest as we
> cannot claim memory!
> xc: error: panic: xg_dom_boot.c:119: xc_dom_boot_mem_init: can't
> allocate low memory for domain: Out of memory
> libxl: error: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init
> failed: Cannot allocate memory
> domainbuilder: detail: xc_dom_release: called
>
> So the claim failed.  The system has enough memory since we're just
> rebooting the same VM.  As a work around, I added sleep(1) + retry,
> which works.
>
> The curious part is the memory allocation.  For d2 to d5, we have:
> domainbuilder: detail: range: start=0x0 end=0xf0000000
> domainbuilder: detail: range: start=0x100000000 end=0x1af000000
> xc: detail: PHYSICAL MEMORY ALLOCATION:
> xc: detail:   4KB PAGES: 0x0000000000000000
> xc: detail:   2MB PAGES: 0x00000000000006f8
> xc: detail:   1GB PAGES: 0x0000000000000003
>
> But when we have to retry the claim for d6, there are no 1GB pages used:
> domainbuilder: detail: range: start=0x0 end=0xf0000000
> domainbuilder: detail: range: start=0x100000000 end=0x1af000000
> domainbuilder: detail: HVM claim failed! attempt 0
> xc: detail: PHYSICAL MEMORY ALLOCATION:
> xc: detail:   4KB PAGES: 0x0000000000002800
> xc: detail:   2MB PAGES: 0x0000000000000ce4
> xc: detail:   1GB PAGES: 0x0000000000000000
>
> But subsequent reboots for d7 and d8 go back to using 1GB pages.
>
> Does the change in memory allocation stick out to anyone?
>
> Unfortunately, I don't have insight into what the failing test is doing.
>
> Xen doesn't seem set up to track the claim across reboot.  Retrying
> the claim works in our scenario since we have a controlled configuration.

This looks to me like a known phenomenon.  Ages back, a change was made
in how Xen scrubs memory, from being synchronous in domain_kill(), to
being asynchronous in the idle loop.

The consequence being that, on an idle system, you can shutdown and
reboot the domain faster, but on a busy system you end up trying to
allocate the new domain while memory from the old domain is still dirty.

It is a classic example of a false optimisation, which looks great on an
idle system only because the idle CPUs are swallowing the work.

This impacts the ability to find a 1G aligned block of free memory to
allocate a superpage with, and by the sounds of it, claims (which
predate this behaviour change) aren't aware of the "to be scrubbed"
queue and fail instead.

I thought OpenXT had a revert of this.  IIRC it was considered a
material regression in being able to know when a domain has gone away.

~Andrew