Re: [Xen-devel] [PATCH 3/6] xen/pvshim: identity pin shim vCPUs to pCPUs

[George Dunlap <dunlapg@xxxxxxxxx>]

On Thu, Jan 25, 2018 at 9:14 AM, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> On Wed, Jan 24, 2018 at 06:03:28PM +0000, George Dunlap wrote:
>> On Wed, Jan 17, 2018 at 9:48 AM, Roger Pau Monne <roger.pau@xxxxxxxxxx> 
>> wrote:
>> > Since VCPUOP_{up/down} already identity pins vCPU hotplug to pCPU
>> > hotplug also pin the vCPUs to the pCPUs in the scheduler. This prevent
>> > vCPU migration and should improve performance.
>> >
>> > While there also use __cpumask_set_cpu instead of cpumask_set_cpu,
>> > there's no need to use the locked variant.
>> >
>> > Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
>>
>> Sorry, I just realized this -- we already have a way to pin a VM 1:1
>> -- d->is_pinned should do what you want here without having to
>> special-case the pvshim.
>>
>> It seems like something like the attached might be better (compile-tested 
>> only).
>
> I haven't tested the proposed patch, but there's a peculiarity of the
> shim that I think makes this approach invalid.
>
> When Xen is booted in shim mode the APs are never started, which means
> that dom0_cpus only contains the BSP, and that alloc_vcpu is always
> going to be called with cpu == 0. This in turn means that
> sched_init_vcpu is also always called with cpu_id == 0, and if
> is_pinned is set it would force all vCPUs to be pinned to the BSP
> AFAICT.

Right, I see -- dom0 may not actually be running on pcpus 0-N (and in
theory there may be more vcpus than pcpus), so the code goes through
and pins them one-by-one based on what cpus it actually has, rather
than using the vcpu_id directly.

So it looks like you want to bring up cpus in Xen only when the guest
brings them up.  So in shim mode, you only bring up the BSP.  You want
all possible "dom0" (i.e. L2) vcpus *created* at boot time, and you
also want them pinned to their respective "pcpus" (L1 vcpus).

But you can't call alloc_vcpu() with the appropriate "pcpuid", because
then sched_init_vcpu() will set v->processor equal to a pcpu which
isn't up yet.  So you set dom0_cpus to contain only cpu0, so that
alloc_vcpu() will always be called with cpu 0; and then special-case
the affinity so that when it *does* come up, the affinity is already
set.

Is that about right?

What about setting the hard affinity in pv_shim_cpu_up() instead?
(You don't need to set the soft affinity, as the hard affinity will
override it.)

-George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel