Re: [PATCH] x86/pod: Do not fragment PoD memory allocations

[George Dunlap <George.Dunlap@xxxxxxxxxx>]

> On Jan 31, 2021, at 6:13 PM, Elliott Mitchell <ehem+xen@xxxxxxx> wrote:
> 
> On Thu, Jan 28, 2021 at 10:42:27PM +0000, George Dunlap wrote:
>> 
>>> On Jan 28, 2021, at 6:26 PM, Elliott Mitchell <ehem+xen@xxxxxxx> wrote:
>>> type = "hvm"
>>> memory = 1024
>>> maxmem = 1073741824
>>> 
>>> I suspect maxmem > free Xen memory may be sufficient.  The instances I
>>> can be certain of have been maxmem = total host memory *7.
>> 
>> Can you include your Xen version and dom0 command-line?
> 
>> This is on staging-4.14 from a month or two ago (i.e., what I happened to 
>> have on a random test  box), and `dom0_mem=1024M,max:1024M` in my 
>> command-line.  That rune will give dom0 only 1GiB of RAM, but also prevent 
>> it from auto-ballooning down to free up memory for the guest.
>> 
> 
> As this is a server, not a development target, Debian's build of 4.11 is
> in use.  Your domain 0 memory allocation is extremely generous compared
> to mine.  One thing which is on the command-line though is
> "watchdog=true".

staging-4.14 is just the stable 4.14 branch which our CI loop tests before 
pushing to stable-4.14, which is essentially tagged 3 times a year for point 
releases.  It’s quite stable.  I’ll give 4.11 a try if I get a chance.

It’s not clear from your response — are you allocating a fixed amount to dom0?  
How much is it?  In fact, probably the simplest thing to do would be to attach 
the output of `xl info` and `xl dmesg`; that will save a lot of potential 
future back-and-forth.

1GiB isn’t particularly generous if you’re running a large number of guests.  
My understanding is that XenServer now defaults to 4GiB of RAM for dom0.

> I've got 3 candidates which presently concern me:ble:
> 
> 1> There is a limited range of maxmem values where this occurs.  Perhaps
> 1TB is too high on your machine for the problem to reproduce.  As
> previously stated my sample configuration has maxmem being roughly 7
> times actual machine memory.

In fact I did a number of binary-search-style experiments to try to find out 
boundary behavior.  I don’t think I did 7x memory, but I certainly did 2x or 3x 
host memory, and the exact number you gave that caused you problems.  In all 
cases for me, it either worked or failed with a cryptic error message (the 
specific message depending on whether I had fixed dom0 memory or autoballooned 
memory).

> 2> Between issuing the `xl create` command and the machine rebooting a
> few moments of slow response have been observed.  Perhaps the memory
> allocator loop is hogging processor cores long enough for the watchdog to
> trigger?

I don’t know the balloon driver very well, but I’d hope it yielded pretty 
regularly.  It seems more likely to me that your dom0 is swapping due to low 
memory / struggling with having to work with no file cache.  Or the OOM killer 
is doing its calculation trying to figure out which process to shoot?  

> 3> Perhaps one of the patches on Debian broke things?  This seems
> unlikely since nearly all of Debian's patches are either strictly for
> packaging or else picks from Xen's main branch, but this is certainly
> possible.

Indeed, I’d consider that unlikely.  Some things I’d consider more likely to 
cause the difference:

1. The amount of host memory (my test box had only 6GiB)

2. The amount of memory assigned to dom0 

3. The number of other VMs running in the background

4. A difference in the version of Linux (I’m also running Debian, but 
deban-testing)

5. A bug in 4.11 that was fixed by 4.14.

If you’re already allocating a fixed amount of memory to dom0, but it’s 
significantly less than 1GiB, the first thing I’d try is increasing that to 
1GiB.  Also make sure that you’re specifying a ‘max’ for dom0 memory: If you 
simply put `dom0_mem=X`, dom0 will start with X amount of memory, but allocate 
enough frame tables such that it could balloon up to the full host memory if 
requested.  (And frame tables are not free.)  `dom0_mem=X,max=X` will cause 
dom0 to only make frame tables for X memory.  (At least, so I’m guessing; I 
haven’t checked.)

If that doesn’t work, please include the output of `xl info` and `xl dmesg`; 
that will give us a lot more information to work with.

Peace,
 -George