WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] RE: Live migration fails due to c/s 20627

To: Andre Przywara <andre.przywara@xxxxxxx>
Subject: RE: [Xen-devel] RE: Live migration fails due to c/s 20627
From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Date: Wed, 16 Dec 2009 08:54:01 -0800 (PST)
Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx, kurt.hackel@xxxxxxxxxx, "Dugger, Donald D" <donald.d.dugger@xxxxxxxxx>, "Xu, Dongxiao" <dongxiao.xu@xxxxxxxxx>, Fraser <keir.fraser@xxxxxxxxxxxxx>, "Nakajima, Jun" <jun.nakajima@xxxxxxxxx>, Keir, "Zhang, Xiantao" <xiantao.zhang@xxxxxxxxx>
Delivery-date: Wed, 16 Dec 2009 08:55:22 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4B28FFA3.7060300@xxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Hi Andre --

> But is it useful to emulate RDTSCP? I see two use cases for this 

Thanks for your support!

> Dan, can you summarize the usage of RDTSCP emulation in PV? 
> Honestly I got lost in all these threads..

Me too :-)

In PV, the "pvcpuid" bit is not set so guest OS's that use
the proper PV access method to cpuid will believe that
the hardware does not support rdtscp.  Since cpuid is
unprivileged, apps running in PV domains may check the
bit and use rdtscp.  In this case, TSC_AUX should
contain 0.  If this domain is saved/restored/migrated
to a machine that does not support the rdtscp, the
instruction is emulated.

If tsc_mode=3, rdtscp is handled specially.

See xen-unstable.hg/docs/misc/tscmode.txt for more info.

Thanks,
Dan

> -----Original Message-----
> From: Andre Przywara [mailto:andre.przywara@xxxxxxx]
> Sent: Wednesday, December 16, 2009 8:41 AM
> To: Dan Magenheimer
> Cc: Jeremy Fitzhardinge; Xu, Dongxiao; xen-devel@xxxxxxxxxxxxxxxxxxx;
> Kurt Hackel; Dugger, Donald D; Keir Fraser; Nakajima, Jun; Zhang,
> Xiantao
> Subject: Re: [Xen-devel] RE: Live migration fails due to c/s 20627
> 
> 
> Dan Magenheimer wrote:
> >> -----Original Message-----
> >> From: Jeremy Fitzhardinge [mailto:jeremy@xxxxxxxx]
> >> Sent: Tuesday, December 15, 2009 11:26 AM
> >> To: Xu, Dongxiao
> >> Cc: Dan Magenheimer; Keir Fraser; 
> xen-devel@xxxxxxxxxxxxxxxxxxx; Kurt
> >> Hackel; Dugger, Donald D; Nakajima, Jun; Zhang, Xiantao
> >> Subject: Re: Live migration fails due to c/s 20627
> >>
> >>
> >> On 12/15/2009 09:24 AM, Xu, Dongxiao wrote:
> >>> If CPU has rdtsc but no rdtscp, then the VM exec control 
> bit in VMCS
> >>> won't be turned on. Therefore if rdtscp instruction runs, 
> >> it will encounter
> >>> invalid op code directly but no VMEXIT.
> >>>    
> >> Ah, right.  You'd need to make that particular illegal 
> >> instruction vmexit.
> > 
> > Or make ALL illegal instructions vmexit, decode, if rdtscp
> > emulate it, else vmenter again.
> But is it useful to emulate RDTSCP? I see two use cases for this 
> instruction:
> 1) NUMA aware malloc:
> You need to know the current node number _quickly_ to use the right 
> bucket to take the memory from. You do not even want using a 
> syscall for 
> this, that's why getcpu in Linux is implemented as a vsyscall either 
> using RDTSCP or LSL. If you emulate this, this will need a 
> few thousand 
> cycles.
> 2) Making sure TSC values are consistent:
> By looking at the core ID you learn whether two consecutive 
> RDTSCPs are 
> from the same core and are thus reliable. If you loose a few thousand 
> cycles with emulation, than the whole purpose of doing the 
> RDTSCPs is in 
> question, as your results would be spoiled due to the overhead.
> 
> These two issues are the main reason I refrained from implementing 
> RDTSCP virtualization some months ago, as even virtualizing them 
> introduces a slight overhead (MSR save/restore). As software seems to 
> cope with not having this instruction (and using the perfectly 
> virtualized lsl instruction, for instance), I thought the 
> benefit would 
> not justify the effort.
> 
> Dan, can you summarize the usage of RDTSCP emulation in PV? 
> Honestly I 
> got lost in all these threads..
> 
> 
> Regards,
> Andre.
> 
> -- 
> Andre Przywara
> AMD-Operating System Research Center (OSRC), Dresden, Germany
> Tel: +49 351 448 3567 12
> ----to satisfy European Law for business letters:
> Advanced Micro Devices GmbH
> Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
> Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
> Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> Registergericht Muenchen, HRB Nr. 43632
> 
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel