[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Poor HVM performance with 8 vcpus


  • To: Tim Deegan <Tim.Deegan@xxxxxxxxxx>
  • From: Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
  • Date: Wed, 07 Oct 2009 14:32:56 +0200
  • Cc: Ian Pratt <Ian.Pratt@xxxxxxxxxxxxx>, James Harper <james.harper@xxxxxxxxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Wed, 07 Oct 2009 05:33:24 -0700
  • Domainkey-signature: s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:Message-ID:Date:From:Organization: User-Agent:MIME-Version:To:CC:Subject:References: In-Reply-To:X-Enigmail-Version:Content-Type: Content-Transfer-Encoding; b=ucWiUCtwZl7CvMsQTdJAsdpJyXSVAVhBhEhXUPuJHW+wzf+OR87roG/y S5Om4SLbxXbS1IHQ6edjDGlfCgI7SlD50labC6JPthQ9bo9z9mY5CyALY JNcjZneSorh4p4eWEfqPWvnEIpVXaiIQTbYMW6jMxc12DfWnq8pvkvRJt D+HY3SjX50OOW4VlQWG0GFnwX2rMzc6s71LcNeRS3W46ybwm+Lo43xT8D cK+w3Pvtp1+gjTWjb4TiIk8GM2KR7;
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

Tim Deegan wrote:
> At 10:40 +0100 on 07 Oct (1254912041), Juergen Gross wrote:
>> calls to shadow_alloc              438     427     424     480     436     
>> 422
>> number of shadow pages in use     2765    2151    2386    2509    4885    
>> 1391
>> calls to shadow_free               168     132     185     144     181     
>> 105
>> calls to shadow_fault            65271   69132   60495   53756   73363   
>> 52449
>> shadow_fault fast path n/p        7347    8081    6713    6134    8521    
>> 6112
>> shadow_fault fast path error        14      12      15       3      13      
>> 11
>> shadow_fault really guest fault  24004   25723   22815   19709   27049   
>> 19190
>> shadow_fault emulates a write     1045     949    1018     995    1015     
>> 901
>> shadow_fault fast emulate          424     361     449     348     387     
>> 314
>> shadow_fault fixed fault         32503   34264   29624   26689   36641   
>> 26096
>> calls to shadow_validate_gl2e      875     748     917     731     795     
>> 667
>> calls to shadow_validate_gl3e      481     456     443     491     489     
>> 446
>> calls to shadow_validate_gl4e      104      97      95     112     105      
>> 95
>> calls to shadow_hash_lookup    2109654 2203254 2228896 2245849 2164727 
>> 2309059
>> shadow hash hit in bucket head 2012828 2111164 2161113 2177591 2104099 
>> 2242458
>> shadow hash misses                 851     840     841     910     852     
>> 838
>> calls to get_shadow_status     2110031 2202828 2228769 2246689 2164213 
>> 2309241
>> calls to shadow_hash_insert        438     436     428     481     437     
>> 430
>> calls to shadow_hash_delete        168     150     185     154     202     
>> 128
>> shadow removes write access        335     324     329     385     330     
>> 336
>> shadow writeable: linux high       130     139     152     155     138     
>> 149
>> shadow writeable: sl1p           14508   15402   12961   11823   16474   
>> 11472
>> shadow writeable brute-force       205     185     177     230     192     
>> 187
>> shadow unshadows for fork/exit       9      12      12      12      18      
>> 12
>> shadow unshadows a page             10      13      13      13      19      
>> 13
>> shadow walks guest tables       647527  727336  649397  646601  659655  
>> 621289
>> shadow checks gwalk                526     544     535     550     614     
>> 554
>> shadow flush tlb by rem wr perm    235     233     229     268     238     
>> 237
>> shadow emulates invlpg           14688   15499   14604   12630   16627   
>> 11370
>> shadow OOS fixup adds            14467   15335   13059   11840   16624   
>> 11339
>> shadow OOS unsyncs               14467   15335   13058   11840   16624   
>> 11339
>> shadow OOS evictions               566     449     565     369     589     
>> 336
>> shadow OOS resyncs               14510   15407   12964   11828   16478   
>> 11481
>>
>> I don't think the "shadow writable brute-force" is the problem.
>> get_shadow_status seems to be a more critical candidate.
> 
> get_shadow_status is a simple hash lookup to find the shadow of a frame;
> it's expected to happen multiple times per pagefault.  Even so those
> numbers look high.  ~10k guest PT walks per CPU per second, each causing
> 3-4 shadow hash lookups.  That's much higher than the number of
> pagefaults.  
> 
> I take it you reset the performance counters at the start of that run?

Yes.

> 
> Are there any other numbers (outside the shadow stats) that are up
> around 600k/cpu?

No. All other counts are below 100k.

> 
> I wonder whether this is caused by pagetables changing under out feet in
> the shadow fault handler -- in order to avoid taking the shadow lock too
> often, the fault handler walks the pagetables first, then takes the lock
> and double-checks its work.   If the other cpus are aggressively writing
> to the pagetables that this CPU is running on that could cause the
> pagefault handler to spin, locking and unlocking the shadow lock as it
> goes.   I would expect to see the OOS unsyncs number much higher if that
> was the case though.  
> 
> Can you try the attached patch and see if it makes any difference?

Not really. Still 4 vcpus waiting at the shadow lock, performance counters as
follows (test ran with 8 vcpus this time, performance was 2% higher than
without patch, but uncertainty is in the same order).
I copied only the counters for 1 cpu, as the others are similar. Counters are
smaller as before, because performance was less as with 6 vcpus and lock
profiling costs about 20% performance with this lock load (the previous
measurement had no lock profiling support).

calls to shadow_alloc               700
number of shadow pages in use      1986
calls to shadow_free                 48
calls to shadow_fault             33508
shadow_fault fast path n/p         3785
shadow_fault fast path error         10
shadow_fault really guest fault   11290
shadow_fault emulates a write      1313
shadow_fault fast emulate           257
shadow_fault fixed fault          16853
calls to shadow_validate_gl2e       604
calls to shadow_validate_gl3e       786
calls to shadow_validate_gl4e       179
calls to shadow_hash_lookup     1079773
shadow hash hit in bucket head  1030265
shadow hash misses                 1258
calls to get_shadow_status      1079233
calls to shadow_hash_insert         700
calls to shadow_hash_delete          48
shadow removes write access         650
shadow writeable: linux high        624
shadow writeable: sl1p             7459
shadow writeable brute-force         26
shadow unshadows for fork/exit        1
shadow unshadows a page               1
shadow walks guest tables        393663
shadow checks gwalk                1251
shadow flush tlb by rem wr perm     439
shadow emulates invlpg             6212
shadow OOS fixup adds              7330
shadow OOS unsyncs                 7328
shadow OOS evictions                175
shadow OOS resyncs                 7459


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 636 47950
Fujitsu Technolgy Solutions               e-mail: juergen.gross@xxxxxxxxxxxxxx
Otto-Hahn-Ring 6                        Internet: ts.fujitsu.com
D-81739 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.