Re: [Xen-devel] [PATCH 3/6] xen/pvshim: identity pin shim vCPUs to pCPUs

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, Jan 25, 2018 at 02:03:35PM +0000, George Dunlap wrote:
> On Thu, Jan 25, 2018 at 9:14 AM, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> > On Wed, Jan 24, 2018 at 06:03:28PM +0000, George Dunlap wrote:
> >> On Wed, Jan 17, 2018 at 9:48 AM, Roger Pau Monne <roger.pau@xxxxxxxxxx> 
> >> wrote:
> >> > Since VCPUOP_{up/down} already identity pins vCPU hotplug to pCPU
> >> > hotplug also pin the vCPUs to the pCPUs in the scheduler. This prevent
> >> > vCPU migration and should improve performance.
> >> >
> >> > While there also use __cpumask_set_cpu instead of cpumask_set_cpu,
> >> > there's no need to use the locked variant.
> >> >
> >> > Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> >>
> >> Sorry, I just realized this -- we already have a way to pin a VM 1:1
> >> -- d->is_pinned should do what you want here without having to
> >> special-case the pvshim.
> >>
> >> It seems like something like the attached might be better (compile-tested 
> >> only).
> >
> > I haven't tested the proposed patch, but there's a peculiarity of the
> > shim that I think makes this approach invalid.
> >
> > When Xen is booted in shim mode the APs are never started, which means
> > that dom0_cpus only contains the BSP, and that alloc_vcpu is always
> > going to be called with cpu == 0. This in turn means that
> > sched_init_vcpu is also always called with cpu_id == 0, and if
> > is_pinned is set it would force all vCPUs to be pinned to the BSP
> > AFAICT.
> 
> Right, I see -- dom0 may not actually be running on pcpus 0-N (and in
> theory there may be more vcpus than pcpus), so the code goes through
> and pins them one-by-one based on what cpus it actually has, rather
> than using the vcpu_id directly.
> 
> So it looks like you want to bring up cpus in Xen only when the guest
> brings them up.  So in shim mode, you only bring up the BSP.  You want
> all possible "dom0" (i.e. L2) vcpus *created* at boot time, and you
> also want them pinned to their respective "pcpus" (L1 vcpus).
> 
> But you can't call alloc_vcpu() with the appropriate "pcpuid", because
> then sched_init_vcpu() will set v->processor equal to a pcpu which
> isn't up yet.  So you set dom0_cpus to contain only cpu0, so that
> alloc_vcpu() will always be called with cpu 0; and then special-case
> the affinity so that when it *does* come up, the affinity is already
> set.
> 
> Is that about right?

Yes, I think so.

> What about setting the hard affinity in pv_shim_cpu_up() instead?

That would mean setting it up each time the CPU is brought up. Seems
easier to set only once during domain build and forget about it.

> (You don't need to set the soft affinity, as the hard affinity will
> override it.)

That seems fine to me. I more or less guessed that, but didn't look
that carefully.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel