Re: domU reboot claim failed

[Jan Beulich <jbeulich@xxxxxxxx>]

On 11.09.2025 23:20, Jason Andryuk wrote:
> Thanks, everyone.
> 
> On 2025-09-10 17:57, Andrew Cooper wrote:
>> On 10/09/2025 7:58 pm, Jason Andryuk wrote:
>>> Hi,
>>>
>>> We're running Android as a guest and it's running the Compatibility
>>> Test Suite.  During the CTS, the Android domU is rebooted multiple times.
>>>
>>> In the middle of the CTS, we've seen reboot fail.  xl -vvv shows:
>>> domainbuilder: detail: Could not allocate memory for HVM guest as we
>>> cannot claim memory!
>>> xc: error: panic: xg_dom_boot.c:119: xc_dom_boot_mem_init: can't
>>> allocate low memory for domain: Out of memory
>>> libxl: error: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init
>>> failed: Cannot allocate memory
>>> domainbuilder: detail: xc_dom_release: called
>>>
>>> So the claim failed.  The system has enough memory since we're just
>>> rebooting the same VM.  As a work around, I added sleep(1) + retry,
>>> which works.
>>>
>>> The curious part is the memory allocation.  For d2 to d5, we have:
>>> domainbuilder: detail: range: start=0x0 end=0xf0000000
>>> domainbuilder: detail: range: start=0x100000000 end=0x1af000000
>>> xc: detail: PHYSICAL MEMORY ALLOCATION:
>>> xc: detail:   4KB PAGES: 0x0000000000000000
>>> xc: detail:   2MB PAGES: 0x00000000000006f8
>>> xc: detail:   1GB PAGES: 0x0000000000000003
>>>
>>> But when we have to retry the claim for d6, there are no 1GB pages used:
>>> domainbuilder: detail: range: start=0x0 end=0xf0000000
>>> domainbuilder: detail: range: start=0x100000000 end=0x1af000000
>>> domainbuilder: detail: HVM claim failed! attempt 0
>>> xc: detail: PHYSICAL MEMORY ALLOCATION:
>>> xc: detail:   4KB PAGES: 0x0000000000002800
>>> xc: detail:   2MB PAGES: 0x0000000000000ce4
>>> xc: detail:   1GB PAGES: 0x0000000000000000
>>>
>>> But subsequent reboots for d7 and d8 go back to using 1GB pages.
>>>
>>> Does the change in memory allocation stick out to anyone?
>>>
>>> Unfortunately, I don't have insight into what the failing test is doing.
>>>
>>> Xen doesn't seem set up to track the claim across reboot.  Retrying
>>> the claim works in our scenario since we have a controlled configuration.
>>
>> This looks to me like a known phenomenon.  Ages back, a change was made
>> in how Xen scrubs memory, from being synchronous in domain_kill(), to
>> being asynchronous in the idle loop.
>>
>> The consequence being that, on an idle system, you can shutdown and
>> reboot the domain faster, but on a busy system you end up trying to
>> allocate the new domain while memory from the old domain is still dirty.
>>
>> It is a classic example of a false optimisation, which looks great on an
>> idle system only because the idle CPUs are swallowing the work.
>>
>> This impacts the ability to find a 1G aligned block of free memory to
>> allocate a superpage with, and by the sounds of it, claims (which
>> predate this behaviour change) aren't aware of the "to be scrubbed"
>> queue and fail instead.
> 
> Claims check total_avail_pages and outstanding_claims.  It looks like 
> free_heap_pages() sets PGC_need_scrub and then increments 
> total_avail_pages.  But then it's not getting through the accounting far 
> enough to stake a claim?
> 
> Also free_heap_page() looks like it's trying to merge chunks - I thought 
> that would handle larger allocations.  Are they not truly usable until 
> they've been scrubbed, which leads to the lack of 1GB pages?
> 
> Clearly I need to learn more here.

I rather expect this then may not be scrubbing related, but domain cleanup
hasn't progressed quickly enough for the earlier instance.

Jan