This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


RE: [Xen-devel] unnecessary VCPU migration happens again

To: "Emmanuel Ackaouy" <ack@xxxxxxxxxxxxx>, "Xu, Anthony" <anthony.xu@xxxxxxxxx>
Subject: RE: [Xen-devel] unnecessary VCPU migration happens again
From: "Petersson, Mats" <Mats.Petersson@xxxxxxx>
Date: Thu, 7 Dec 2006 11:52:26 +0100
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, xen-ia64-devel <xen-ia64-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Thu, 07 Dec 2006 02:52:41 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <20061207103755.GA21653@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AccZ69BblHKyLR3UQlOflTqI2lVtggAAarcw
Thread-topic: [Xen-devel] unnecessary VCPU migration happens again

> -----Original Message-----
> From: Emmanuel Ackaouy [mailto:ack@xxxxxxxxxxxxx] 
> Sent: 07 December 2006 10:38
> To: Xu, Anthony
> Cc: Petersson, Mats; xen-devel@xxxxxxxxxxxxxxxxxxx; xen-ia64-devel
> Subject: Re: [Xen-devel] unnecessary VCPU migration happens again
> On Thu, Dec 07, 2006 at 11:37:54AM +0800, Xu, Anthony wrote:
> > >From this logic, the migration happens frequently if the 
> numbers VCPU
> > is less than the number of logic CPU.
> This logic is designed to make better use of a partially idle
> system by spreading work across sockets and cores before
> co-scheduling them. It won't come into play if there are no
> idle execution units.
> Note that __csched_running_vcpu_is_stealable() will trigger a
> migration only when the end result would be strictly better
> than the current situation. Once the system is balanced, it
> will not bounce VCPUs around.
> > That I want to highlight is,
> > 
> > When HVM VCPU is executing IO operation,
> > This HVM VCPU is blocked by HV, until this IO operation
> > is emulated by Qemu. Then HV wakes up this HVM VCPU.
> > 
> > While PV VCPU will not be blocked by PV driver.
> > 
> > 
> > I can give below senario.
> > 
> > There are two sockets, two core per socket.
> > 
> > Assume, dom0 is running on socket1 core1,
> >  vti1 is runing on socket1 core2,
> > Vti 2 is runing on socket2 core1,
> > Socket2 core2 is idle.
> > 
> > If vti2 is blocked by IO operation, then socket2 core1 is idle,
> > That means two cores in socket2 are idle,
> > While dom0 and vti1 are running on two cores of socket1,
> > 
> > Then scheduler will try to spread dom0 and vti1 on these 
> two sockets.
> > Then migration happens. This is no necessary.
> Argueably, if 2 unrelated VCPUs are runnable on a dual socket
> host, it is useful to spread them across both sockets. This
> will give each VCPU more achievable bandwidth to memory.
> What I think you may be argueing here is that the scheduler
> is too aggressive in this action because the VCPU that blocked
> on socket 2 will wake up very shortly, negating the host-wide
> benefits of the migration when it does while still maintaining
> the costs.
> There is a tradeoff here. We could try being less aggressive
> in spreading stuff over idle sockets. It would be nice to do
> this with a greater understanding of the tradeoff though. Can
> you share more information, such as benchmark perf results,
> migration statistics, or scheduler traces?

I don't know if I've understood this right or not, but I believe the
penalty for switching from one core (or socket) to another is higher on
IA64 than on x86. I'm not an expert on IA64, but I remember someone at
the Xen Summit saying something to that effect - I think it was
something like executing a bunch of code to flush the TLB's or some

> Emmanuel.

Xen-devel mailing list