| |
|
 |
|
 |
|
|
|
| |
|
|
xen-devel
Re: [Xen-devel] Poor HVM performance with 8 vcpus
Tim Deegan wrote:
> At 10:40 +0100 on 07 Oct (1254912041), Juergen Gross wrote:
>> calls to shadow_alloc 438 427 424 480 436
>> 422
>> number of shadow pages in use 2765 2151 2386 2509 4885
>> 1391
>> calls to shadow_free 168 132 185 144 181
>> 105
>> calls to shadow_fault 65271 69132 60495 53756 73363
>> 52449
>> shadow_fault fast path n/p 7347 8081 6713 6134 8521
>> 6112
>> shadow_fault fast path error 14 12 15 3 13
>> 11
>> shadow_fault really guest fault 24004 25723 22815 19709 27049
>> 19190
>> shadow_fault emulates a write 1045 949 1018 995 1015
>> 901
>> shadow_fault fast emulate 424 361 449 348 387
>> 314
>> shadow_fault fixed fault 32503 34264 29624 26689 36641
>> 26096
>> calls to shadow_validate_gl2e 875 748 917 731 795
>> 667
>> calls to shadow_validate_gl3e 481 456 443 491 489
>> 446
>> calls to shadow_validate_gl4e 104 97 95 112 105
>> 95
>> calls to shadow_hash_lookup 2109654 2203254 2228896 2245849 2164727
>> 2309059
>> shadow hash hit in bucket head 2012828 2111164 2161113 2177591 2104099
>> 2242458
>> shadow hash misses 851 840 841 910 852
>> 838
>> calls to get_shadow_status 2110031 2202828 2228769 2246689 2164213
>> 2309241
>> calls to shadow_hash_insert 438 436 428 481 437
>> 430
>> calls to shadow_hash_delete 168 150 185 154 202
>> 128
>> shadow removes write access 335 324 329 385 330
>> 336
>> shadow writeable: linux high 130 139 152 155 138
>> 149
>> shadow writeable: sl1p 14508 15402 12961 11823 16474
>> 11472
>> shadow writeable brute-force 205 185 177 230 192
>> 187
>> shadow unshadows for fork/exit 9 12 12 12 18
>> 12
>> shadow unshadows a page 10 13 13 13 19
>> 13
>> shadow walks guest tables 647527 727336 649397 646601 659655
>> 621289
>> shadow checks gwalk 526 544 535 550 614
>> 554
>> shadow flush tlb by rem wr perm 235 233 229 268 238
>> 237
>> shadow emulates invlpg 14688 15499 14604 12630 16627
>> 11370
>> shadow OOS fixup adds 14467 15335 13059 11840 16624
>> 11339
>> shadow OOS unsyncs 14467 15335 13058 11840 16624
>> 11339
>> shadow OOS evictions 566 449 565 369 589
>> 336
>> shadow OOS resyncs 14510 15407 12964 11828 16478
>> 11481
>>
>> I don't think the "shadow writable brute-force" is the problem.
>> get_shadow_status seems to be a more critical candidate.
>
> get_shadow_status is a simple hash lookup to find the shadow of a frame;
> it's expected to happen multiple times per pagefault. Even so those
> numbers look high. ~10k guest PT walks per CPU per second, each causing
> 3-4 shadow hash lookups. That's much higher than the number of
> pagefaults.
>
> I take it you reset the performance counters at the start of that run?
Yes.
>
> Are there any other numbers (outside the shadow stats) that are up
> around 600k/cpu?
No. All other counts are below 100k.
>
> I wonder whether this is caused by pagetables changing under out feet in
> the shadow fault handler -- in order to avoid taking the shadow lock too
> often, the fault handler walks the pagetables first, then takes the lock
> and double-checks its work. If the other cpus are aggressively writing
> to the pagetables that this CPU is running on that could cause the
> pagefault handler to spin, locking and unlocking the shadow lock as it
> goes. I would expect to see the OOS unsyncs number much higher if that
> was the case though.
>
> Can you try the attached patch and see if it makes any difference?
Not really. Still 4 vcpus waiting at the shadow lock, performance counters as
follows (test ran with 8 vcpus this time, performance was 2% higher than
without patch, but uncertainty is in the same order).
I copied only the counters for 1 cpu, as the others are similar. Counters are
smaller as before, because performance was less as with 6 vcpus and lock
profiling costs about 20% performance with this lock load (the previous
measurement had no lock profiling support).
calls to shadow_alloc 700
number of shadow pages in use 1986
calls to shadow_free 48
calls to shadow_fault 33508
shadow_fault fast path n/p 3785
shadow_fault fast path error 10
shadow_fault really guest fault 11290
shadow_fault emulates a write 1313
shadow_fault fast emulate 257
shadow_fault fixed fault 16853
calls to shadow_validate_gl2e 604
calls to shadow_validate_gl3e 786
calls to shadow_validate_gl4e 179
calls to shadow_hash_lookup 1079773
shadow hash hit in bucket head 1030265
shadow hash misses 1258
calls to get_shadow_status 1079233
calls to shadow_hash_insert 700
calls to shadow_hash_delete 48
shadow removes write access 650
shadow writeable: linux high 624
shadow writeable: sl1p 7459
shadow writeable brute-force 26
shadow unshadows for fork/exit 1
shadow unshadows a page 1
shadow walks guest tables 393663
shadow checks gwalk 1251
shadow flush tlb by rem wr perm 439
shadow emulates invlpg 6212
shadow OOS fixup adds 7330
shadow OOS unsyncs 7328
shadow OOS evictions 175
shadow OOS resyncs 7459
Juergen
--
Juergen Gross Principal Developer Operating Systems
TSP ES&S SWE OS6 Telephone: +49 (0) 89 636 47950
Fujitsu Technolgy Solutions e-mail: juergen.gross@xxxxxxxxxxxxxx
Otto-Hahn-Ring 6 Internet: ts.fujitsu.com
D-81739 Muenchen Company details: ts.fujitsu.com/imprint.html
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
|
|
| |
|
Home