Re: [Xen-devel] [PATCH RFC v1 42/74] sched/null: skip vCPUs on the waitqueue that are blocked

[Dario Faggioli <dfaggioli@xxxxxxxx>]

On Fri, 2018-01-12 at 10:45 +0000, Roger Pau Monné wrote:
> On Fri, Jan 12, 2018 at 10:54:03AM +0100, Dario Faggioli wrote:
>
> > Err... yes. BTW, either there are a couple of typos in the above
> > paragraph, or it's me that can't read it well. Anyway, just to be
> > clear, if we have 4 pCPUs, and 6 VMs, with 1 vCPU each, this might
> > be
> > the situation:
> > 
> > CPU0 <-- d1v0
> > CPU1 <-- d2v0
> > CPU2 <-- d3v0
> > CPU3 <-- d4v0
> > 
> > Waitqueue: d5v0,d6v0
> > 
> > Then, if d2 leaves/dies/etc, leaving CPU1 idle, d5v0 is picked up
> > from
> > the waitqueue and assigned to CPU1.
> 
> I think the above example is not representative of what happens
> inside
> of the shim, 
>
Indeed it's not. I was just trying to clarify, via an example, George's
explanation of how null works in general.

> since there's only one domain that runs on the shim, so
> the picture is something like:
> 
> CPU0 <-- d1v0
> CPU1 <-- d1v1
> 
> waitqueue: d1v2 (down), d1v3 (down)
> 
Right. So, how about we change this in such a way that d1v2 and d1v3,
since they're offline, won't end up in the waitqueue?

> Then if the guest brings up another vCPU, let's assume it's vCPU#3
> pCPU#3 will be bring up form the shim PoV, and the null scheduler
> will
> assign the first vCPU on the waitqueue:
> 
> CPU0 <-- d1v0
> CPU1 <-- d1v1
> CPU3 <-- d1v2 (down)
> NULL <-- d1v3 (up)
> 
> Hence d1v2 which is still down will get assigned to CPU#3, and d1v3
> which is up won't get assigned to any pCPU, and hence won't run.
> 
Exactly. While, if d1v2 and d1v3 were not in the waitqueue, while
offline, at all, whould would (should) happen is:

- CPU3 comes online ("in" the shim)
- CPU3 stays idle, as there's nothing in the waitqueue
- d1v3 comes online and is added to the shim's null scheduler
- as CPU3 does not have any vCPU assigned, d1v3 is assigned to it

> > Mmm, wait. In case of a domain which specifies both maxvcpus and
> > curvcpus, how many vCPUs does the domain in which the shim run?
> 
> Regardless of the values of maxvcpus and curvcpus PV guests are
> always
> started with only the BSP online, and then the guest itself brings up
> other vCPUs.
> 
> In the shim case vCPU hotplug is tied to pCPU hotplug, so everytime
> the guest hotplugs or unplugs a vCPU the shim does exactly the same
> with it's CPUs.
> 
Sure, what I was asking was much rather this: if the guest config file
has "maxvcpus=4;vcpus=1", at the end of domain creation, and before any
`xl vcpu-set' or anything that would bring online other guest vCPU,
what's the output of `vl vcpu-list'. :-)

Anyway, I think you've answered to this below.

> > I'm not sure how an offline vCPU can end up there... but maybe I
> > need
> > to look at the code better, with the shim use case in mind.
> > 
> > Anyway, I'm fine with checks that prevent offline vCPUs to be
> > assigned
> > to either pCPUs (like, the CPUs of L0 Xen) or shim's vCPUs (so, the
> > CPUs of L1 Xen). I'm less fine with rescheduling everyone at every
> > wakeup.
> 
> So using the scenario from before:
> 
> CPU0 <-- d1v0
> CPU1 <-- d1v1
> 
> waitqueue: d1v2 (down), d1v3 (down)
> 
> Guest decided to hotplug vCPU#2, and hence the shim first hotplugs
> CPU#2, but at the point CPU2 is added to the pool of CPUs vCPU2 is
> still not up, hence we get the following:
> 
> CPU0 <-- d1v0
> CPU1 <-- d1v1
> CPU2 <-- NULL
> 
> waitqueue: d1v2 (down), d1v3 (down)
> 
> Then d1v2 is brought up, but since the null scheduler doesn't react
> to
> wakeup the picture stays the same:
> 
> CPU0 <-- d1v0
> CPU1 <-- d1v1
> CPU2 <-- NULL
> 
> waitqueue: d1v2 (up), d1v3 (down)
> 
> And d1v2 doesn't get scheduled.
> 
> Hope this makes sense :)
> 
Yeah, and I see that it works.

What I'm saying is that I'd prefer, instead than having the null
scheduler reacting to wakeups of vCPUs in the waitqueue, to avoid
having the offline vCPUs in the waitqueue all together.

At which point, when d1v2 hotplug happens, there has to be a
null_vcpu_insert() (or something equivalent), to which the null
scheduler should react already.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel