Increasing domain memory beyond initial maxmem

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase
domain memory beyond initial maxmem, but I hit few issues.

A little context: domains in Qubes OS start with rather little memory
(400MB by default) but maxmem set higher (4GB by default). Then, there is
qmemman daemon, that adjust balloon targets for domains, based on (among
other things) demand reported by the domains themselves. There is also a
little swap, to mitigate qmemman latency (few hundreds ms at worst).
This initial memory < maxmmem in case of PVH / HVM makes use of PoD
which I'm trying to get rid of. But also, IIUC Linux will waste some
memory for bookkeeping based on maxmem, not actually usable memory.

First issue: after using `xl mem-max`, `xl mem-set` still refuses to
increase memory more than initial maxmem. That's because xl mem-max does
not update 'memory/static-max' xenstore node. This one is easy to work
around.

Then, the actual hotplug fails on the domU side with:

[   50.004734] xen-balloon: vmemmap alloc failure: order:9, 
mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), 
nodemask=(null),cpuset=/,mems_allowed=0
[   50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 
5.16.15-1.37.fc32.qubes.x86_64 #1
[   50.004792] Call Trace:
[   50.004799]  <TASK>
[   50.004808]  dump_stack_lvl+0x48/0x5e
[   50.004821]  warn_alloc+0x162/0x190
[   50.004832]  ? __alloc_pages+0x1fa/0x230
[   50.004842]  vmemmap_alloc_block+0x11c/0x1c5
[   50.004856]  vmemmap_populate_hugepages+0x185/0x519
[   50.004868]  vmemmap_populate+0x9e/0x16c
[   50.004878]  __populate_section_memmap+0x6a/0xb1
[   50.004890]  section_activate+0x20a/0x278
[   50.004901]  sparse_add_section+0x70/0x160
[   50.004911]  __add_pages+0xc3/0x150
[   50.004921]  add_pages+0x12/0x60
[   50.004931]  add_memory_resource+0x12b/0x320
[   50.004943]  reserve_additional_memory+0x10c/0x150
[   50.004958]  balloon_thread+0x206/0x360
[   50.004968]  ? do_wait_intr_irq+0xa0/0xa0
[   50.004978]  ? decrease_reservation.constprop.0+0x2e0/0x2e0
[   50.004991]  kthread+0x16b/0x190
[   50.005001]  ? set_kthread_struct+0x40/0x40
[   50.005011]  ret_from_fork+0x22/0x30
[   50.005022]  </TASK>

Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45

After the above, `free` reports correct size (1GB in this case), but
that memory seems to be unusable really. "used" is kept low, and soon
OOM-killer kicks in.

I know the initial 400MB is not much for a full Linux, with X11 etc. But
I wouldn't expect it to fail this way when _adding_ memory.

I've tried also with initial 800MB. In this case, I do not get "alloc
failure" any more, but monitoring `free`, the extra memory still doesn't
seem to be used.

Any ideas?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature