Re: [Xen-devel] [PATCH] xen/pvhvm: Support more than 32 VCPUs when migrating (v3).

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Fri, Nov 13, 2015 at 02:49:22PM +0000, Ian Campbell wrote:
> On Fri, 2015-11-13 at 09:42 -0500, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 12, 2015 at 04:40:06PM +0000, Ian Campbell wrote:
> > > On Fri, 2015-07-10 at 14:57 -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Jul 10, 2015 at 02:37:46PM -0400, Konrad Rzeszutek Wilk
> > > > wrote:
> > > > > When Xen migrates an HVM guest, by default its shared_info can
> > > > > only hold up to 32 CPUs. As such the hypercall
> > > > > VCPUOP_register_vcpu_info was introduced which allowed us to
> > > > > setup per-page areas for VCPUs. This means we can boot PVHVM
> > > > > guest with more than 32 VCPUs. During migration the per-cpu
> > > > > structure is allocated freshly by the hypervisor (vcpu_info_mfn
> > > > > is set to INVALID_MFN) so that the newly migrated guest
> > > > > can make an VCPUOP_register_vcpu_info hypercall.
> > > > > 
> > > > > Unfortunatly we end up triggering this condition in Xen:
> > > > > /* Run this command on yourself or on other offline VCPUS. */
> > > > >  if ( (v != current) && !test_bit(_VPF_down, &v->pause_flags) )
> > > > > 
> > > > > which means we are unable to setup the per-cpu VCPU structures
> > > > > for running vCPUS. The Linux PV code paths make this work by
> > > > > iterating over every vCPU with:
> > > > > 
> > > > >  1) is target CPU up (VCPUOP_is_up hypercall?)
> > > > >  2) if yes, then VCPUOP_down to pause it.
> > > > >  3) VCPUOP_register_vcpu_info
> > > > >  4) if it was down, then VCPUOP_up to bring it back up
> > > > > 
> > > > > But since VCPUOP_down, VCPUOP_is_up, and VCPUOP_up are
> > > > > not allowed on HVM guests we can't do this. However with the
> > > > > Xen git commit f80c5623a126afc31e6bb9382268d579f0324a7a
> > > > > ("xen/x86: allow HVM guests to use hypercalls to bring up vCPUs"")
> > > > 
> > > > <sigh> I was in my local tree which was Roger's 'hvm_without_dm_v3'
> > > > looking at patches and spotted this - and thought it was already in!
> > > > 
> > > > Sorry about this patch - and please ignore it until the VCPU_op*
> > > > can be used by HVM guests.
> > > 
> > > FYI I just tripped over this while implementing ARM save/restore (in
> > > that I
> > > couldn't figure out HTF HVM VCPUs > MAX_LEGACY_VCPUS were getting their
> > > vcpu_info re-registered, which turns out to be because they aren't...).
> > > 
> > > ARM also lack the VCPU_up/down/is_up hypercalls. My plan there is
> > > simply to
> > > use on_each_cpu to do it, I can get away with this on ARM because the
> > > necessary infra (IPIs etc) are provided by the h/w virt platform (i.e.
> > > look
> > > native) so there is no reliance on Xen infra being fully up.
> > > 
> > > Not sure if that is also true of x86/PVHVM but thought I would mention
> > > it
> > > in case it seemed preferable to you.
> > 
> > Yes, but we have a hard-limit of 32 CPUs on 'HVM' guests and on the
> > shared_info structure. Hence to go above that we need to use the VCPU_op
> > calls.
> 
> I think you mean s/HVM/Linux PVHVM/? Because HVM_MAX_VCPUS in the
> hypervisor is 128.

Thank you. I confused the CPU support with event channels support.

The issue was the shared page and the events - the shared
page by default only has enough slots for 32 CPUs. Anything above that
and we need to use the fancy VCPUOP hypercalls. And that is exactly
what we end up doing during bootup. But if we do save/restore we get to
the issue that only one hypercall is allowed on HVM: VCPUOP_register_vcpu_info

And that hypercall can only be called on for vCPUS which are offline
or if we (the VCPU) is calling it. However the restore path brings
every CPU from CPU0. We could send an IPI to the other CPU such that
it would call VCPUOP_register_vcpu_info.. but IPIs are using events and
the events are not working past 32CPUs unless you call 
VCPUOP_register_vcpu_info.

> 
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel