This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU

To: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough
From: Dante Cinco <dantecinco@xxxxxxxxx>
Date: Fri, 19 Nov 2010 16:13:33 -0800
Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx>, Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>, mathieu.desnoyers@xxxxxxxxxx, Andrew Thomas <andrew.thomas@xxxxxxxxxx>, Konrad Wilk <konrad.wilk@xxxxxxxxxx>, Keir Fraser <keir@xxxxxxx>, Chris Mason <chris.mason@xxxxxxxxxx>
Delivery-date: Fri, 19 Nov 2010 16:14:51 -0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=zBjP6OdJE32ln4uQQ0sFjIDvBKje+mEhjX8RFCV6jmk=; b=BSkN6HRE1WEWw1cLdVuiMzP9v0hlTk1MeJ8L9AtogAu1UOZqcOKU/JfsPMhEI6PLFE WvJu63YCCw5JHNJ6ZThkoSyLXmJNqe3/PoCxUDWI5deXHSvlh6q/LW8l1YTM9QQLFbTy En+0/2SlbFZCGE0YkIOnR9LpP59QAZBUPpAE0=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=jsSNqkOtZbGDRbpqv2oabPFY+7rRO44czdAcwC5nxAcCBNfakPlTBwRnEFxEj1wUsA ZyTsJMgXAJWhwfkRMxez8RRB0o0l31EVX9GSjQJQON6MXL7aV6UIn5watqLlbTAP0E34 jvXXZyMrDfSCRUKS14R1h4OFqlrgmpPRdymoE=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <81120886-968e-42ac-b70c-124345d901c8@default>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C90C6B36.A992%keir@xxxxxxx> <81120886-968e-42ac-b70c-124345d901c8@default>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Fri, Nov 19, 2010 at 2:36 PM, Dan Magenheimer
<dan.magenheimer@xxxxxxxxxx> wrote:
>> From: Keir Fraser [mailto:keir@xxxxxxx]
>> Sent: Friday, November 19, 2010 10:58 AM
>> To: Dante Cinco; Jeremy Fitzhardinge
>> Cc: Xen-devel; mathieu.desnoyers@xxxxxxxxxx; Chris Mason; Andrew
>> Thomas; Konrad Rzeszutek Wilk
>> Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2
>> pvops domU kernel with PCI passthrough
>> On 19/11/2010 17:52, "Dante Cinco" <dantecinco@xxxxxxxxx> wrote:
>> > How do I check if rdtsc emulation is on? Does 'xm debug-keys s' do
>> it?
>> >
>> > (XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch
>> > input to DOM0)
>> > (XEN) TSC marked as reliable, warp = 0 (count=2)
>> > (XEN) dom1: mode=0,ofs=0xca6f68770,khz=2666017,inc=1
>> > (XEN) No domains have emulated TSC
>> TSC emulation is not enabled.
> I *think* "No domains have emulated TSC" will be printed
> if there are no domains other than dom0 currently running,
> so this may not be definitive.

The pvops domU was running when I captured that Xen console output and
I also looked at /var/log/xen/xend.log and saw 'tsc_mode 0' without
explicitly setting tsc_mode in the domain's cfg file.

> Also note that tsc_mode=0 means "do the right thing for
> this hardware platform" but, if the domain is saved/restored
> or live-migrated, TSC will start being emulated. See
> tscmode.txt in xen/Documentation for more detail.

We have not done any save/restore on domU.

> Lastly, I haven't tested this code in quite some time,
> the code for PV and HVM is different, and I've never
> tested it with xl, only with xm.  So bitrot is possible,
> though hopefully unlikely.
> Thanks,
> Dan

pvclock_clocksource_read is no longer the top symbol (was 28% of the
CPU samples) in the latest xenoprofile report. I had mistakenly
attributed the huge I/O performance gain (from 119k IOPS to 209k IOPS)
to the act of killing ntpd but that was not the case. In fact, the
performance gain was due to turning off lock stat. I had enabled lock
stat in the kernel to try to track down the lock-associated symbols in
the profile report. I had forgotten that I had turned off lock stat
(echo 0 >/proc/sys/kernel/lock_stat) just before I killed ntpd. When I
disabled lock stat in the kernel, I was able to get 209k IOPS without
killing ntpd.

The latest xenoprofile report doesn't even have
pvclock_clocksource_read in the top 10. All the I/O processing in domU
(domID=1) is done in our kernel driver modules so domain1-modules is
expected to be at the top of the list.

CPU: Intel Architectural Perfmon, speed 2665.97 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               app name                 symbol name
542839   17.2427  domain1-modules          domain1-modules
378968   12.0375
domain1-kernel           xen_spin_unlock
250342    7.9518  vmlinux-
206585    6.5620  xen-syms-4.1-unstable    domain1-xen
123021    3.9076
domain1-kernel           lock_release
103703    3.2940
domain1-kernel           __lock_acquire
100973    3.2073  domain1-xen-unknown      domain1-xen-unknown
94449     3.0001
domain1-kernel           hypercall_page
67145     2.1328  xen-syms-4.1-unstable    domain1-xen
64460     2.0475
domain1-kernel           xen_spin_trylock
62415     1.9825
domain1-kernel           xen_restore_fl_direct
51822     1.6461
domain1-kernel           native_read_tsc
45901     1.4580  vmlinux-
44398     1.4103
domain1-kernel           debug_locks_off
42191     1.3402
domain1-kernel           find_next_bit
41913     1.3313
domain1-kernel           do_raw_spin_lock
41424     1.3158
domain1-kernel           lock_acquire
39275     1.2475  xen-syms-4.1-unstable    domain1-xen

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>