Re: [Xen-devel] Can't always start 32 bit domains after 64 bit domains

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 21/11/16 10:05, Jan Beulich wrote:
>>>> On 21.11.16 at 10:45, <srn@xxxxxxxxx> wrote:
>> On 11/21/2016 12:20 AM, Jan Beulich wrote:
>>>>>> On 19.11.16 at 22:22, <srn@xxxxxxxxx> wrote:
>>>> My current understanding is that on a server with more than 168GiB
>>>> of memory, I should still be able to around 128GiB of 32-bit PV
>>>> domUs, regardless of what order the domUs are started in.
>>> You don't clarify what you base this understanding of yours on; I don't
>>> think this is the case. What exact memory will be allocated for 64-bit
>>> guests isn't very predictable. In particular, the preference of allocating
>>> 1Gb pages as long as available may result in allocations eating into the
>>> low 128Gb of memory earlier than you expect. You may want to analyze
>>> system state by looking at debug key output at relevant points in time.
>> Because whenever I read about the 168GiB limit and 32 bit domains there was 
>> no mention that the initial 128GiB might not all be available if there
>> were also 64 bit domains. Given how little visibility there typically is 
>> into where memory is physically allocated, there is no way to know this
>> doesn't work as a normal user until it doesn't. Are you saying all Xen users 
>> have to be developers too?
> No, I'm not saying this, but please note that you didn't post the
> question on xen-users, but xen-devel.
>
>> In my opinion this is a bug until/unless it's very clearly documented that 
>> it's not supported behavior to mix 32 and 64 bit PV domains when there is
>> more than 168GiB RAM, or maybe just period if that's too complicated to 
>> explain to people. I don't know where to find that documentation. It seems
>> like https://wiki.xenproject.org/wiki/Xen_Project_Release_Features might be 
>> the right place, I'm not sure. If you know of a better place I'd love to 
>> know.
> I view this as a bug too, fwiw. Me previously bringing up the topic
> didn't result in any activity though, presumably because 32-bit
> guests are being considered rare enough these days.
>
>>> Back in the xend days someone here had invented a (crude) mechanism
>>> to set aside memory for 32-bit PV domains, but I don't think dealing with
>>> this situation in xl has ever seen any interest.
>> If I wanted to add that, where would it go?
> I don't know, that's a question for xl folks I guess.

IIRC, given no other constraints, the Xen heap allocation allocates from
the top down to help this exact case.

I suspect that libxl's preference towards NUMA allocation of domains
interferes with this, by adding a NUMA constraints to memory allocations
for 64bit PV guests.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel