Re: [Xen-devel] [PATCH 19/24] xen: credit2: soft-affinity awareness in load balancing

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Fri, 2016-09-02 at 12:46 +0100, anshul makkar wrote:

Hey, Anshul,

Thanks for having a look at the patch!

> On 17/08/16 18:19, Dario Faggioli wrote:
> > 
> > --- a/xen/common/sched_credit2.c
> > +++ b/xen/common/sched_credit2.c
> > 
> > + * Basically, if a soft-affinity is defined, the work done by a
> > vcpu on a
> > + * runq to which it has higher degree of soft-affinity, is
> > considered
> > + * 'lighter' than the same work done by the same vcpu on a runq to
> > which it
> > + * has smaller degree of soft-affinity (degree of soft affinity is
> > <= 1). In
> > + * fact, if soft-affinity is used to achieve NUMA-aware
> > scheduling, the higher
> > + * the degree of soft-affinity of the vcpu to a runq, the greater
> > the probability
> > + * of accessing local memory, when running on such runq. And that
> > is certainly\
> > + * 'lighter' than having to fetch memory from remote NUMA nodes.
> Do we ensure that while defining soft-affinity for a vcpu, NUMA 
> architecture is considered. If not, then this whole calculation can
> go 
> wrong and have negative impact on performance.
> 
Defining soft-affinity after topology is what we do by default, just
not here in Xen: we do it in toolstack (in libxl, to be precise).

NUMA aware scheduling is indeed the most obvious use case for all this
--and, in fact that's why we configure things in such a way in higher
layers-- but the mechanism is, at the Xen level, flexible enough to be
used for any purpose that the user may find interesting.

> Degree of affinity to runq will give good result if the affinity to 
> pcpus has been chosen after due consideration ..
>
At this level, 'good result' means 'making sure that a vcpu runs for as
much time as possible on a pcpu to which it has soft-affinity'. Whether
that is good or not for performance (or for any other aspect or
metric), it's not this algorithm's job to determine.

Note that things are exactly the same for hard-affinity/pinning, or for
weights. In fact, Xen won't stop one to, say, pin 128 vcpu all to pcpu
3. This will deeply suck, but it's the higher layers' will (fault?) and
Xen should just comply to that.

> > + * If there is no soft-affinity, load_balance() (actually,
> > consider()) acts
> > + * as follows:
> > + *
> > + *  - D = abs(Li - Lj)
> If we are consider absolute of Li -Lj, how will we know which runq
> has 
> less workload which, I think, is an essential parameter for load 
> balancing. Am I missing something here ?
>
What we are aiming for is making the queues more balanced, which means
we want the difference between their load to be smaller than how it is
when the balancing start. As far as that happens, we don't care which
loads goes down and which one goes up, as far as the final result is a
smaller load delta.

> > + *  - consider pushing v from I to J:
> > + *     - D' = abs(Li - lv - (Lj + lv))   (from now, abs(x) == |x|)
> > + *     - if (D' < D) { push }
> > + *  - consider pulling k from J to I:
> > + *     - D' = |Li + lk - (Lj - lk)|
> > + *     - if (D' < D) { pull }
> For both push and pull we are checking (D` < D) ?
>
Indeed. And that's because of the abs(). :-)


Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel