Re: [Xen-devel] PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote:
> On 07/28/2015 06:29 AM, Juergen Gross wrote:
> > On 07/27/2015 04:09 PM, Dario Faggioli wrote:
> >> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
> >>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
> >>
> >>>> So, just to check if I'm understanding is correct: you'd like to add an
> >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> >>>> code, to hide the direct interaction with CPUID.
> >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> >>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
> >>>
> >>> Sort of, yes.
> >>>
> >>> I just wouldn't add it, as it is already existing (more or less). It
> >>> can deal right now with AMD and Intel, we would "just" have to add Xen.
> >>>
> >> So, having gone through the rest of the thread (so far), and having
> >> given a fair amount o thinking to this, I really think that something
> >> like this would be a good thing to have in Linux.
> >>
> >> Of course, it's not that my opinion on where should be in Linux counts
> >> that much! :-D   Nevertheless, I wanted to make it clear that, while
> >> skeptic at the beginning, I now think this is (part of) the way to go,
> >> as I said and explained in my reply to George.
> >
> > I think it's time to obtain some real numbers.
> >
> > I'll make some performance tests on a big machine (4 sockets, 60 cores,
> > 120 threads) regarding topology information:
> >
> > - bare metal
> > - "random" topology (like today)
> > - "simple" topology (all vcpus regarded as equal)
> > - "real" topology with all vcpus pinned
> >
> > This should show:
> >
> > - how intrusive would the topology patch(es) be?
> > - what is the performance impact of a "wrong" scheduling data base
> 
> On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch
> (see attachment). I did 5 kernel builds in each environment:
> 
> make clean
> time make -j 120
> 
> The first result of the 5 runs was always omitted as it would have to
> build up buffer caches etc. The Xen cases were all done in dom0, pinning
> of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter
> of the hypervisor.
> 
> Here are the results (everything in seconds):
> 
>                      elapsed   user   system
> bare metal:            100    5770      805
> "random" topology:     283    6740    20700
> "simple" topology:     290    6740    22200
> "real" topology:       185    7800     8040
> 
> As expected bare metal is the best. Next is "real" topology with pinned
> vcpus (expected again - but system time already factor of 10 up!).
> What I didn't expect is: "random" is better than "simple" topology. I
> could test some other topologies (e.g. everything on one socket, or even
> on one core), but I'm not sure this makes sense. I didn't check the
> exact topology result of the "random" case, maybe I'll do that tomorrow
> with another measurement.
>
> BTW: the topology hack is working, as each cpu is shown to have a
> sibling count of 1 in /proc/cpuinfo.
> 
Hey, just a 'wild' idea, as a possible explanation (or direction for
further investigations) on why 'simple' si actually worse than 'random'.

Could it be that, by setting up the topology like in 'simple', we create
more work for the Linux scheduler, and hence incur in more overhead? Or,
looking at thing the other way round, which is it the topology that
impose the less overhead, in terms of load balancing work, to the Linux
scheduler?

I think we should identify and try it as, in the absence of any vNUMA
topology being specified, that's what we want, as our flat topology (and
with vNUMA, that's what we want within virtual nodes).


I don't really know, right now, whether such topology would be 'all
cores, no SMT siblings', or 'all siblings of each others', or 'all
sockets, no core/SMT siblings'... This requires inspecting the
scheduler's and scheduling domain's code, which I can do, but not today,
as I won't be working. If you (or anyone) think it's worth and actually
have a look, let me/us know your findings. If not, I'll do it myself
tomorrow.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel