WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] [PATCH] x86/hvm/pmtimer: improving scalability of virtual ti

To: Keir Fraser <keir@xxxxxxx>, Jan Beulich <JBeulich@xxxxxxxxxx>
Subject: [Xen-devel] [PATCH] x86/hvm/pmtimer: improving scalability of virtual time update on Xen 4.0.0
From: Song Xiang <classicxsong@xxxxxxxxx>
Date: Wed, 17 Nov 2010 21:39:05 +0000
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, Haibo Chen <oldseawave@xxxxxxxxx>
Delivery-date: Wed, 17 Nov 2010 05:43:37 -0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :content-type:mime-version:subject:date:cc:x-mailer; bh=9Tg1qgx5ZmhnT7hEayQ3bp7EvnYox+qBx6K7hfd/uEI=; b=VItBwrHumPBxpy6mOpANwjbaybrGgjzllfYYs4nOu/tf0qO70zUL+eGMchyxlpdOjD zOW+8rSyKmaf+SErOpq7JgMumb28kxsi5piOzrc/vWiuzvyQkJCUkkFvOrmLMKnt47o5 dwmABu7tYrZx+e0wSyS30gGDAr6OvbOSSyNmg=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:content-type:mime-version:subject:date:cc :x-mailer; b=OdS+OXdI99e3YdRN2X9WJB7ZhLU/l6vUwPjwxNvkzL6KU2sYsIVMi9UxKJVb2ithZK ytDCmwaS/elOpnEPXwSalgE6Lla2T89Xw3dTdAQVhx7MWd0m3wX46bzhCDe1nGARaoyQ 9jI6ph6VBGfyLwQFqTvm+EeQGH8eti+YcPWW4=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
From: Xiang Song <xiangsong@xxxxxxxxxxxx>

This provides a patch to arch/x86/hvm/pmtimer.c for Xen 4.0.0 to mitigate the heavy contention on handle_pmt_io when running a HVM configured with many cores (e.g., 32 cores). As the virtual time of a domain must be fresh, there should be someone updating it, periodically. But it is not necessary to let a VCPU update the virtual time when another one has been updating it. Thus the update can be skipped when the VCPU finds someone else is updating the virtual time. So every time a VCPU invoke handle_pmt_io to update the current domain's virtual time, it will first try to acquire the pmtimer lock. If it succeeds, it will update the virtual time. Otherwise, it can skip the update, waits for the pmtimer lock holder to finish updating the virtual time and returns the updated time.

Signed-off-by: Xiang Song <xiangsong@xxxxxxxxxxxx>
---
xen/arch/x86/hvm/pmtimer.c |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)

We evaluate this patch using a guest HVM configured with 32 cores on a 48-core HP machine. (8 * 6 AMD 2.4GHz Opteron chips). We use Linux (Debian GNU/Linux 5.0, kernel version 2.6.35-rc5) as guest HVM OS.  Each VCPU of the guest HVM is pinned on a unique physical core.  The guest HVM is allocated with 16GBytes memory with HAP enabled. 
We use 7 application benchmarks to evaluate the patch, and run all of them in memory using a ramfs.

The percentage of CPU time spent on spin_lock inside handle_pmt_io under 32 cores configuration on HVM on the original Xen is as follows: (We use Xenoprof to collect the execution info)
original Xen
gmake  32.56%
phoenix -- histogram 55.04%
phoenix -- wordcount  78.94%
phoenix -- linear_regression 15.89%
specjvm -- compress 21.79%
specjvm -- crypto AES  21.35%
specjvm -- XML validation 42.73%


It is obvious that all applications have heavy contention in handle_pmt_io. After applying the patch, The percentage of CPU time spent on _spin_barrier inside handle_pmt_io under 32 cores configuration is much lower than that on original Xen:
patched Xen
gmake  2.20%
phoenix -- histogram 12.25%
phoenix -- wordcount  17.04%
phoenix -- linear_regression 4.45%
specjvm -- compress 2.45%
specjvm -- crypto AES  2.46%
specjvm -- XML validation 4.67%


After applying the patch, we get speedups for all 7 applications. The performance of 7 applications using 1-core configuration and 32-core configuration on original Xen and patched Xen, and the speedup after using this patch are shown as follows:
1-core 32-core
original Xen patched Xen speedup original Xen patched Xen speedup
gmake (sec) 91.2 89.2 2.24% 1824.3 1778 2.60%
phoenix -- histogram (sec) 9.5 9.15 3.83% 32.79 28.67 14.37%
phoenix -- wordcount  (sec) 206.23 201.07 2.57% 279.22 249.53 11.90%
phoenix -- linear_regression  (sec) 2.57 2.5 2.80% 6.06 5.89 2.89%
specjvm -- compress (ops/min) 27.43 28.6 4.27% 774.37 906.0366667 17.00%
specjvm -- crypto AES  (ops/min) 8.92 9.32 4.48% 209.55 245.5166667 17.16%
specjvm -- XML validation  (ops/min) 62.55 65.65 4.96% 624.46 780.89 25.05%


diff --git a/xen/arch/x86/hvm/pmtimer.c b/xen/arch/x86/hvm/pmtimer.c
index 48fe26a..587944f 100644
--- a/xen/arch/x86/hvm/pmtimer.c
+++ b/xen/arch/x86/hvm/pmtimer.c
@@ -90,7 +90,7 @@ static void pmt_update_time(PMTState *s)
 
     /* Update the timer */
     curr_gtime = hvm_get_guest_time(s->vcpu);
-    s->pm.tmr_val += ((curr_gtime - s->last_gtime) * s->scale) >> 32;
+    *(volatile uint32_t *)&s->pm.tmr_val = s->pm.tmr_val + (((curr_gtime - s->last_gtime) * s->scale) >> 32);
     s->pm.tmr_val &= TMR_VAL_MASK;
     s->last_gtime = curr_gtime;
     
@@ -206,10 +206,19 @@ static int handle_pmt_io(
     
     if ( dir == IOREQ_READ )
     {
-        spin_lock(&s->lock);
-        pmt_update_time(s);
-        *val = s->pm.tmr_val;
-        spin_unlock(&s->lock);
+        /*
+         * if acquired the PMTState lock then update the time
+         * else other vcpu is updating it ,it should be up to date.
+         */
+        if (spin_trylock(&s->lock)) {
+            pmt_update_time(s);
+            *val = s->pm.tmr_val;
+            spin_unlock(&s->lock);
+        }
+        else {
+            spin_barrier(&s->lock);
+            *val = s->pm.tmr_val;
+        }
         return X86EMUL_OKAY;
     }

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel