Re: [Xen-users] Aoe or iScsi???

[Adi Kriegisch <kriegisch@xxxxxxxx>]

Hi!

> I run bonnie++ like this:
> bonnie++ -d /tmp/ -s 1000 -r 500 -n 1 -x 1 -u root |  bon_csv2txt > test.txt
just checking: your storage server has 500MB RAM? (-r)

> This is the result:
> 
> Version  1.03c      ------Sequential Output------ --Sequential Input- 
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
> %CP
> bacula-selbet 1000M 53004  98 189796  37 97783  17 62844  99 1505552  99 +++++
[SNIP]
> It's tell something?
Ja, your storage system can handle ~190MB/s sequential write. This means
you will not get full peak performance to your clients as one gigabit
interface is limited with 120MB/s.
Your write speed (1,15GB/s) shows that you misspecified RAM size on your
bonnie commandline because this is _WAY_ beyond what your disks are able to
handle. (Good SATA disks will give you above 100MB/s read speed. Reading at
that speed hints at 15 or more disks; the limit there is definitely bus
speed and administrative overhead.)

What you are really interested in (or should be) are IOPS (Input Output
Operations per Second): A typical server or workstation no matter if
virtual or 'real' does a mixture between sequential and random I/O.
Every server you run has its own partition on your storage backend. Just to
get a better idea of what I am talking about consider the following:
Every virtual machine does a sequential file read. What does that mean on
the storage backend? -- There are 13 files being read at 13 different
positions at the same time. This is a (close to) random I/O workload. Disk
heads are flying around to satisfy all requests. No way you will be close
to any high MB/s value: your disks are doing random I/O.
Measuring sequential peak performances on network storage is pointless for
this very reason. (People on this list were suggesting to do that just to
verify your disk subsystem works fine.)
To get an idea of what performance you might expect, you can do the
following:
1. calculate IOPS that you might expect. You may use one of the online
   calculators that are available[1].
   This begins with calculating IOPS per disk where you might need to
   consult your vendor's datasheet or lookup the disks here[2]. You'll
   immediately notice that SAS disks offer twice or more IOPS than SATA
   drives.
   When calculating IOPS you need to specify a workload as well. This means
   specify the read/write ratio. Average fileservers have around 80% read
   and 20% write. Read and write operations differ in the latency they
   have: The more latency a request has the fewer requests can be handled
   per second. (This is also the reason why local storage will always bring
   more IOPS than network storage: network transport simply adds to latency.)
2. measure the IOPS you get. I personally prefer using FIO[3] which is
   readily available in Debian. FIO is fully configurable; there are
   however some reasonable examples which you might use:
   /usr/share/doc/fio/examples/iometer-file-access-server mimiks a typical
   file server workload with 80% read. The IOPS calculator above[1] is only
   able to calculate IOPS for a fixed block size where this workload mixes
   blocksizes from 512byte to 64k. The result in IOPS cannot be directly
   compared. If you want to do so, you need to specify 4k blocks only in the
   config.
   WARNING: Do not use IOMeter itself on linux: it provides incorrect
   results as it cannot use aio on linux and therefor is unable to queue
   requests.
   Using the stock 'iometer-file-access-server' profile you should get the
   something like:
   3 disks/RAID5: 200-250 IOPS
   4 disks/RAID5: 270-320 IOPS
   5 disks/RAID5: 340-390 IOPS
   and so on (for SATA disks with AoE).
3. find the bottleneck in case you're not getting what you can expect.
   Measure IOPS on the storage server with 'iostat 1' ("tps" roughly
   corresponds to IOPS).
   ...ok, writing up how to debug a storage backend will take another
   hour... ask me if necessary.

-- Adi

[1] http://www.wmarow.com/strcalc/
[2] http://www.wmarow.com/strdir/hdd/
[3] http://freshmeat.net/projects/fio

PS: Maybe there should be a wiki page about how to plan and implement a
storage backend for a xen server? -- then others can add their knowledge
more easily.
...and the question pops up every once in a while.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users