This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] [PATCH 0/6] xen,xend,tools: NUMA support for Xen

To: Ian Pratt <m+Ian.Pratt@xxxxxxxxxxxx>
Subject: Re: [Xen-devel] [PATCH 0/6] xen,xend,tools: NUMA support for Xen
From: Ryan Harper <ryanh@xxxxxxxxxx>
Date: Tue, 11 Jul 2006 20:23:13 -0500
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Tue, 11 Jul 2006 18:23:42 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <A95E2296287EAD4EB592B5DEEFCE0E9D57206B@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <A95E2296287EAD4EB592B5DEEFCE0E9D57206B@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.6+20040907i
* Ian Pratt <m+Ian.Pratt@xxxxxxxxxxxx> [2006-07-11 16:28]:
> > > What sort of box are these numbers taken from? If it's not a NUMA
> > > system then the slowdowns are rather poor. We're particularly
> > > interested in not slowing down non-NUMA and small-NUMA (e.g., AMD
> K8)
> > > x86 systems. They are what we really want to see measurements from.
> > 
> > The measurements are taken from a two-way Operton 248 (2.1Ghz) ,
> > small-NUMA.  I agree that there is significant overhead, however, we
> > aren't talking about fast path here; correct me if I'm wrong.   We
> > are only adding overhead to during domain startup.  The end result
> > being we pay for local memory allocation at creation time while
> > benefiting from local memory access for the lifetime of the domain.
> > 
> > I'm going to gather some oprofile data to see if I missed something
> > obvious, but in general I think that having local memory is of greater
> > benefit for the lifetime of a domain than the cost we incur during its
> > creation.
> What do the numbers look like on a 1 node system?

For K8 small numa, I don't have a 1 node system available.

> The shadow mode code potentially churns the page allocator a fair bit.
> It'll be disappointing if we have to add complexity of quicklists etc. 

Yeah, I forgot about shadow mode; good point.

> It does kind of surprise me that the overhead is as high as you've
> measured. In the case where there's memory available in the favoured
> node I'd expect allocation performance to be very similar. 4 times
> slower and worsening for large allocations seems odd -- 0.3 microseconds
> a page is a bit more than I'd expect during back-to-back allocations.
> It's certainly worth trying to understand the overhead a bit more.

I agree.  I'm a little mystified by the overhead as well.  On the larger
system, ballooning up to 23G had something like 11% overhead, which was
more reasonable, though the domain creation tests showed more than 11%
on that system as well.  I'll get the oprofile data and take a look.

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253

Xen-devel mailing list