Re: [Xen-devel] [PATCH 8/9] xen: sched: allow for choosing credit2 runqueues configuration at boot

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Thu, 2015-10-01 at 07:48 +0200, Juergen Gross wrote:
> On 09/29/2015 06:56 PM, Dario Faggioli wrote:
> > In fact, credit2 uses CPU topology to decide how to arrange
> > its internal runqueues. Before this change, only 'one runqueue
> > per socket' was allowed. However, experiments have shown that,
> > for instance, having one runqueue per physical core improves
> > performance, especially in case hyperthreading is available.
> > 
> > In general, it makes sense to allow users to pick one runqueue
> > arrangement at boot time, so that:
> >   - more experiments can be easily performed to even better
> >     assess and improve performance;
> >   - one can select the best configuration for his specific
> >     use case and/or hardware.
> > 
> > This patch enables the above.
> > 
> > Note that, for correctly arranging runqueues to be per-core,
> > just checking cpu_to_core() on the host CPUs is not enough.
> > In fact, cores (and hyperthreads) on different sockets, can
> > have the same core (and thread) IDs! We, therefore, need to
> > check whether the full topology of two CPUs matches, for
> > them to be put in the same runqueue.
> > 
> > Note also that the default (although not functional) for
> > credit2, since now, has been per-socket runqueue. This patch
> > leaves things that way, to avoid mixing policy and technical
> > changes.
> 
> I think you should think about a way to make this parameter a per
> cpupool one instead a system global one. 
>
Believe it or not, I though about this already, and yes, it is in my
plans to make this per-cpupool. However...

> As this will require some
> extra work regarding the tools interface I'd be absolutely fine with
> adding this at a later time, but you should have that in mind when
> setting this up now.
> 
...yes, that was phase II in my mind as well.

So (sorry, but just to make sure I understand), since you said you're
fine with it coming later, are you also fine with this patch, or do you
think some adjustments are necessary, right here, right now, because of
that future plan?

> > --- a/xen/common/sched_credit2.c
> > +++ b/xen/common/sched_credit2.c
> > @@ -82,10 +82,6 @@

> > @@ -194,6 +190,41 @@ static int __read_mostly
> > opt_overload_balance_tolerance = -3;
> >   integer_param("credit2_balance_over",
> > opt_overload_balance_tolerance);
> > 
> >   /*
> > + * Runqueue organization.
> > + *
> > + * The various cpus are to be assigned each one to a runqueue, and
> > we
> > + * want that to happen basing on topology. At the moment, it is
> > possible
> > + * to choose to arrange runqueues to be:
> > + *
> > + * - per-core: meaning that there will be one runqueue per each
> > physical
> > + *             core of the host. This will happen if the
> > opt_runqueue
> > + *             parameter is set to 'core';
> > + *
> > + * - per-socket: meaning that there will be one runqueue per each
> > physical
> > + *               socket (AKA package, which often, but not always,
> > also
> > + *               matches a NUMA node) of the host; This will
> > happen if
> > + *               the opt_runqueue parameter is set to 'socket';
> 
> Wouldn't it be a nice idea to add "per-numa-node" as well?
> 
I think it is.

> This would make a difference for systems with:
> 
> - multiple sockets per numa-node
> - multiple numa-nodes per socket
> 
Yep.

> It might even be a good idea to be able to have only one runqueue in
> small cpupools (again, this will apply only in case you have a per
> cpupool setting instead a global one).
> 
And I agree on this too.

TBH, I had considered these too, and I was thinking to make them happen
in phase II as well. However, they're simple enough to be implemented
now (as in, in v2 of this series), so I think I'll do that.

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel