[Xen-users] domU has better I/O performance than dom0?

[Troels Arvin <troels@xxxxxxxx>]

Hello,

I've done some I/O benchmarks on an RHEL 5.1-based xen setup. The main 
(dom0) server is an x86_64 host with a FC-connected IBM SAN. The guest 
servers are paravirtualized.

I used bonnie++ to stress-test and to try to analyze I/O performance. 
Bonnie++ was run with this command, current working directory being the 
relevant part of the file system:
bonnie++ -n 4 -s 20g -x 5 -u nobody

The chunk size (20g) was much larger than the memory available for the 
dom0/domU (both having ½GB RAM available).
Testing ran for several hours. No stability problems were seen.

Bonnie++ was run on the dom0 and one of the domUs, making sure that the 
involved disks were otherwise idle, or very close to being idle. I.e., a 
quiet (and somewhat slow) part of the SAN was used.

The storage area was made available for the dom0 as a file system on "raw 
device" (no use of logical volume management on the operating system). 
After testing on the dom0, the same device was unmounted, the file system 
was re-created and subsequently allocated to the domU as a "phy"-device; 
the phy-device was then mounted to a suitable mountpoint on the domU.

The tests were run on different times of the day. The dom0 test was 
partly run during work hours where there may have been contention on the 
SAN signalling infrastructure (switch/storage HBAs), although we 
generally believe that we don't have a major bottleneck on the SAN 
optical pathways. The domU test was run during night-time where I/O 
pressure on the SAN infrastructure is probably somewhat lower (although 
various backup and batch jobs make sure that the SAN is never sleeping).

The results were averaged, after filtering out result which seemed 
atypical.

Bonnie wasn't able to detect a difference in the file-creation 
performance, so these values aren't included.

The results:

Host  |     Sequential Output       | Sequential Input  | Random seeks |
      |           (K/sec)           |      (K/sec)      |   (/sec)     |
      | Per Char | Block  | Rewrite | Per Char | Block  |              |
======+=============================+===================+==============+
dom0  |   56739  |  96529 |  46524  |   58346  | 119830 |       94     |
------+----------+--------+---------+----------+--------+--------------+
domU  |   56186  | 112796 |  50178  |   65325  | 202569 |      148     |
======+=============================+===================+==============+
domU  |          |        |         |          |        |              |
gain% |       -1 |     17 |      8  |      12  |     69 |       56     |

In other words: I've found that my domU's I/O performance generally 
surpasses that of my dom0(!).

On a side note: Running mke2fs went much, much faster on the dom0 than on 
the domU. So for this kind of I/O, the pattern seems to break.

Am I just being an ignorant benchmark-idiot, or could this kind of result 
actually be expected and/or explained?

Is bonnie++ a bad storage benchmarking tool? - If so: What else is better?

-- 
Regards,
Troels Arvin <troels@xxxxxxxx>
http://troels.arvin.dk/


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users