WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] [Patch 2 of 2]: PV-domain SMP performance Linux-part

To: Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Subject: Re: [Xen-devel] [Patch 2 of 2]: PV-domain SMP performance Linux-part
From: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Date: Fri, 16 Jan 2009 09:41:39 -0800
Cc: George Dunlap <George.Dunlap@xxxxxxxxxxxxx>, Juergen Gross <juergen.gross@xxxxxxxxxxxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Fri, 16 Jan 2009 09:42:02 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <C59609B4.21084%keir.fraser@xxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C59609B4.21084%keir.fraser@xxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.19 (X11/20090105)
Keir Fraser wrote:
On 16/01/2009 09:36, "Juergen Gross" <juergen.gross@xxxxxxxxxxxxxxxxxxx>
wrote:

The approach taken in Linux is not merely 'yield on spinlock' by the way, it
is 'block on event channel on spinlock' essentially turning a contended
spinlock into a sleeping mutex. I think that is quite different behaviour
from merely yielding, and expecting the scheduler to do something sensible
with your yield request.
Could you explain this a little bit more in detail, please?

Jeremy Fitzhardinge did the implementation for Linux, so I'm cc'ing him in
case he remembers more details than me.

Basically each CPU allocates itself an IPI event channel at start of day.
When a CPU attempts to acquire a spinlock it spins a short while (perhaps a
few microseconds?) and then adds itself to a bitmap stored in the lock
structure (I think, or it might be a linked list of sleepers?). It then
calls SCHEDOP_poll listing its IPI evtchn as its wakeup requirement. When
the lock holder releases the lock it checks for sleepers and if it sees one
then it pings one of them (or is it all of them?) on its event channel, thus
waking it to take the lock.

Yes, that's more or less right. Each lock has a count of how many cpus are waiting for the lock; if its non-zero on unlock, the unlocker kicks all the waiting cpus via IPI. There's a per-cpu variable of "lock I am waiting for"; the kicker looks at each cpu's entry and kicks it if its waiting for the lock being unlocked.

The locking side does the expected "spin for a while, then block on timeout". The timeout is settable if you have the appropriate debugfs option enabled (which also produces quite a lot of detailed stats about locking behaviour). The IPI is never delivered as an event BTW; the locker uses the event poll hypercall to block until the event is pending (this hypercall had some performance problems until relatively recent versions of Xen; I'm not sure which release versions has the fix).

The lock itself is a simple byte spinlock, with no fairness guarantees; I'm assuming (hoping) that the pathological cases that ticket locks were introduced to solve will be mitigated by the timeout/blocking path (and/or less likely in a virtual environment anyway).

I measured a small performance improvement within the domain with this patch (kernbench-type workload), but an overall 10% reduction in system-wide CPU use with multiple competing domains.

   J

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel