[Xen-users] Performance questions.

[Kai Meyer <kai@xxxxxxxxxxxxx>]

I have a problem I'd like to get some input on. Here's my setup:
2x4 core Intel(R) Xeon(R) CPU E5320 @ 1.86GHz
16GB Memory
4x Barracuda 7200.10 SATA 3.0Gb/s 160-GB Hard Drive
4port 3ware 9550SXU-4LP, with the 4 drives in 2 Raid1 mirrors.
Dom-0 Centos 5.1 x86_64
kernel 2.6.18-53.1.6.el5xen

I installed 32 identical "bare metals" Centos 5.1 x86_64 instances. I 
put a php script that simply calculates pi to a certain degree of 
accuracy, and adjusted it so it took about .4 seconds to run on one of 
the Dom-Us. Then from another server, I am using the 'ab' command from 
apache to benchmark performance on the 32 servers simultaniously. Here's 
an example of the command I am running:
ab -n 20 -c 20 http://192.168.1.201/pi.php &

I loop this 32 times, once for each server. It's right around a 
concurrency of 20, and a count of anything higher, when I start getting 
time outs from 'ab'.
apr_poll: The timeout specified has expired (70007)

Out of the batch of 32, I will invariably get between 3 and 16 that 
return this error. If I lower the concurrency to 19, I'll get between 0 
and 5 time outs. 18 and below, I don't get any failures. Over 20, I the 
number of servers that time out do increase, but very slowly.
I'm simply trying to find the threshold or cap that I am hitting so I 
can give a suggestion on how to monitor and manage the highest number of 
virtual machines possible.
What I've tried looking for is a correlation between when the time out 
occurs, and the following list of indicators on both the Dom-0 as well 
as all the Dom-Us.
Interrupts from vmstat
Context Switches from vmstat
CPU usage from vmstat/top/xentop
apache connections from the server-status module
load average

I would prefer to be able to track this sort of behavior from Dom-0. I 
have test results from running various different combinations of virtual 
machines, and concurrency levels from ab, but havn't found the 
underlying cause yet. Any help or pointers on what else to look for 
would be appreciated.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users