Re: [Xen-devel] [PATCH RFC v1 42/74] sched/null: skip vCPUs on the waitqueue that are blocked

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Fri, Jan 12, 2018 at 12:16:47PM +0100, Dario Faggioli wrote:
> On Fri, 2018-01-12 at 10:45 +0000, Roger Pau Monné wrote:
> > On Fri, Jan 12, 2018 at 10:54:03AM +0100, Dario Faggioli wrote:
> >
> > > Err... yes. BTW, either there are a couple of typos in the above
> > > paragraph, or it's me that can't read it well. Anyway, just to be
> > > clear, if we have 4 pCPUs, and 6 VMs, with 1 vCPU each, this might
> > > be
> > > the situation:
> > > 
> > > CPU0 <-- d1v0
> > > CPU1 <-- d2v0
> > > CPU2 <-- d3v0
> > > CPU3 <-- d4v0
> > > 
> > > Waitqueue: d5v0,d6v0
> > > 
> > > Then, if d2 leaves/dies/etc, leaving CPU1 idle, d5v0 is picked up
> > > from
> > > the waitqueue and assigned to CPU1.
> > 
> > I think the above example is not representative of what happens
> > inside
> > of the shim, 
> >
> Indeed it's not. I was just trying to clarify, via an example, George's
> explanation of how null works in general.
> 
> > since there's only one domain that runs on the shim, so
> > the picture is something like:
> > 
> > CPU0 <-- d1v0
> > CPU1 <-- d1v1
> > 
> > waitqueue: d1v2 (down), d1v3 (down)
> > 
> Right. So, how about we change this in such a way that d1v2 and d1v3,
> since they're offline, won't end up in the waitqueue?

Sounds fine. I have to admit this is the first time I play with the
scheduler code, so it's quite likely that whatever you say will seem
OK to me :).

> > Then if the guest brings up another vCPU, let's assume it's vCPU#3
> > pCPU#3 will be bring up form the shim PoV, and the null scheduler
> > will
> > assign the first vCPU on the waitqueue:
> > 
> > CPU0 <-- d1v0
> > CPU1 <-- d1v1
> > CPU3 <-- d1v2 (down)
> > NULL <-- d1v3 (up)
> > 
> > Hence d1v2 which is still down will get assigned to CPU#3, and d1v3
> > which is up won't get assigned to any pCPU, and hence won't run.
> > 
> Exactly. While, if d1v2 and d1v3 were not in the waitqueue, while
> offline, at all, whould would (should) happen is:
> 
> - CPU3 comes online ("in" the shim)
> - CPU3 stays idle, as there's nothing in the waitqueue
> - d1v3 comes online and is added to the shim's null scheduler
> - as CPU3 does not have any vCPU assigned, d1v3 is assigned to it

Yes, that's what I'm aiming for :).

> > So using the scenario from before:
> > 
> > CPU0 <-- d1v0
> > CPU1 <-- d1v1
> > 
> > waitqueue: d1v2 (down), d1v3 (down)
> > 
> > Guest decided to hotplug vCPU#2, and hence the shim first hotplugs
> > CPU#2, but at the point CPU2 is added to the pool of CPUs vCPU2 is
> > still not up, hence we get the following:
> > 
> > CPU0 <-- d1v0
> > CPU1 <-- d1v1
> > CPU2 <-- NULL
> > 
> > waitqueue: d1v2 (down), d1v3 (down)
> > 
> > Then d1v2 is brought up, but since the null scheduler doesn't react
> > to
> > wakeup the picture stays the same:
> > 
> > CPU0 <-- d1v0
> > CPU1 <-- d1v1
> > CPU2 <-- NULL
> > 
> > waitqueue: d1v2 (up), d1v3 (down)
> > 
> > And d1v2 doesn't get scheduled.
> > 
> > Hope this makes sense :)
> > 
> Yeah, and I see that it works.
> 
> What I'm saying is that I'd prefer, instead than having the null
> scheduler reacting to wakeups of vCPUs in the waitqueue, to avoid
> having the offline vCPUs in the waitqueue all together.
> 
> At which point, when d1v2 hotplug happens, there has to be a
> null_vcpu_insert() (or something equivalent), to which the null
> scheduler should react already.

That seems fine to me, I will try to take a look at implementing this.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel