[Xen-devel] [PATCH] Fix performance problems with mprotect()

["Bruce Rogers" <brogers@xxxxxxxxxx>]

While working on a database scaling problem using a SAP database test suite we 
discovered that these enterprise level large shared memory databases are very 
heavy users of mprotect(), to the extent that the performance overhead in 
current Xenolinux  impacts scaling beyond a few cpus quite badly.  A single cpu 
run under Xen has a nominal impact, but scaling out to 8 cpus results in a 
performance of less than 10% of native throughput. Ouch!

The problem was isolated to the relatively high overhead of how mprotect() 
updates ptes, and the lack of efficient batching opportunities as presently 
implemented.  By adding a hypercall which batches pte updates such that 
hardware flags updates (accessed or dirty bits) are not lost, and redoing the 
performance critical portion of mprotect() to take advantage of that, 
performance has improved to where it is essentially equivalent to native 
performance with this SAP database test suite.  I haven't tested other database 
implementations, but I would imagine they would have similar results.

This is obviously quite an improvement and should allow Xen to be deployed for 
more enterprise level workloads.

There are actually 2 new hypercalls included in this patch - one that batches 
flags-only changes to pte's within a single page table, and another which 
allows the full pte to be changed - both "atomically" as described above.

I imagine there are other places in Linux that could benefit from these 
hypercalls, but these patches focus on the mprotect performance problem only.

The hypervisor patch was mostly written and tested with the  3.0.4 era 
hypervisor that ships with SLES 10, and then ported and tested on current 
xen-unstable.
The Linux patch was written and tested on 2.6.16 and made apply to the 2.6.18 
tree without further testing.

Feedback and comments are appreciated.

Signed-off-by: Bruce Rogers <brogers@xxxxxxxxxx>

- Bruce Rogers

Attachment: xen_mprotect_perf.patch
Description: Text document

Attachment: linux_mprotect_perf.patch
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel