WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] 10 million cycles disappearing

To: "Tian, Kevin" <kevin.tian@xxxxxxxxx>, "Xen-Devel (E-mail)" <xen-devel@xxxxxxxxxxxxxxxxxxx>, jbeulich@xxxxxxxxxx
Subject: RE: [Xen-devel] 10 million cycles disappearing
From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Date: Thu, 9 Apr 2009 11:19:06 -0700 (PDT)
Cc:
Delivery-date: Thu, 09 Apr 2009 11:19:54 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <0A882F4D99BBF6449D58E61AAFD7EDD61036A210@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
After looking into this a bit more, it appears to be
only an artifact and a race in my measurement code
(buried in macros where it was non-obvious).

Thanks for the help and sorry for the noise.

P.S. Average of tmem ops including page copy and
compression/decompresion is in the 20K-50K cycle range.

> -----Original Message-----
> From: Tian, Kevin [mailto:kevin.tian@xxxxxxxxx]
> Sent: Wednesday, April 08, 2009 10:47 PM
> To: Dan Magenheimer; Xen-Devel (E-mail); jbeulich@xxxxxxxxxx
> Subject: RE: [Xen-devel] 10 million cycles disappearing
>
>
> In your later test with memory allocation, is it still same case that
> only some samples approach 10M, or the average close to 10M?
> If average cost is high, then xenoprofile could be useful here to
> show hotpot to you.
>
> Thanks,
> Kevin
>
> >From: Dan Magenheimer [mailto:dan.magenheimer@xxxxxxxxxx]
> >Sent: 2009年4月9日 8:47
> >
> >The problem still occurs on latest tip (c/s 19515). :-(
> >
> >
> >> -----Original Message-----
> >> From: Tian, Kevin [mailto:kevin.tian@xxxxxxxxx]
> >> Sent: Tuesday, April 07, 2009 6:33 PM
> >> To: Dan Magenheimer; Xen-Devel (E-mail); jbeulich@xxxxxxxxxx
> >> Subject: RE: [Xen-devel] 10 million cycles disappearing
> >>
> >>
> >> I remember that some previous post (from Jan?) solved some heavy
> >> operation incurred in some special action of writable page
> able. Not
> >> recall the detail, but your description about subop with memory
> >> allocation leads me to that part...
> >>
> >> Thanks,
> >> Kevin
> >>
> >> >From: Dan Magenheimer
> >> >Sent: 2009年4月8日 7:54
> >> >
> >> >I've been seeing a possible performance problem off and on
> >> >and I've spent some time tracking it but haven't made much
> >> >progress and have to give up for now, so I thought I'd at
> >> >least document what I know and see if it sounds familiar
> >> >to anyone.
> >> >
> >> >The problem: Something in Xen seems to periodically take about
> >> >10M cycles.  I think it is an interrupt and I think it is
> >> >taking a lock related to memory allocation and holding
> >> >it for a LONG time (i.e. 10M cycles or close).
> >> >
> >> >I am measuring inside a hypercall using TSC, taking a TSC
> >> >reading at entry to the hypercall code and at exit.  Xen
> >> >is not pre-emptive, so it can't be switching context or
> >> >something, right?  Nearly all of the readings are less
> >> >than 100K cycles, but some samples are "huge" and
> >> >usually at 9M-10M cycles.  Since I am recording the max
> >> >difference between the TSCs, the max "huge" grows over
> >> >a long period of time, but eventually converges close
> >> >to 10M (and this is a 3Ghz processor).  I can see
> >> >it grow using "watch".  And I've NEVER seen a reading
> >> >over 10M.
> >> >
> >> >I am able to disable interrupts and still take
> >> >measurements.  Roughly half of the measurements
> >> >occur when doing a hypercall-subop that does no
> >> >memory allocation and roughly half occur when doing
> >> >a hypercall-subop that DOES do memory allocation.
> >> >With interrupts disabled, the subop that DOES
> >> >memory allocation still asymptotically approaches
> >> >10M.  The one that does NOT do memory allocation,
> >> >stays relatively small.
> >> >
> >> >I'm currently measuring on Xen 3.3.1 but I think I've
> >> >seen similar results on xen-unstable.  A single 2-vcpu
> >> >domain is running (in addition to domain0).
> >> >
> >> >Does any of that sound familiar?  Any smoking guns?
> >> >
> >> >_______________________________________________
> >> >Xen-devel mailing list
> >> >Xen-devel@xxxxxxxxxxxxxxxxxxx
> >> >http://lists.xensource.com/xen-devel
> >> >
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel