[Xen-users] disk I/O problems under load? (Xen-3.4.1/x86_64)
I'm getting problems whenever the load on a system increase, but IMHO
it should be well withing hardware capabilities.
- HP Proliant DL160G5, with a single quadcore E5405, 14GiB RAM, 2x1TB
sata disks Hitachi 7K1000.B on the onboard sata controller (intel
- Xen-3.4.1 64bit hypervisor, compiled from gentoo portage, with
default commandline settings (I just specify the serial console and
- Domain-0 with gentoo's xen-sources 2.6.21 (the xen 2.6.18 tarball
didn't have networking, I think the HP Tigon3 gigabit driver is too
old but hadn't time to look into that
- Domain-0 is using the CFQ i/o scheduler, and works from a software
raid-1, no tickless kernel, HZ=100. It has all the free ram (currently
some 5.x GiB)
- the rest of the disks is also mirrored in a raid-1 device, and I use
LVM2 on top of that
- 6x paravirt 64bit DomU with 2.6.29-gentoo-r5 kernel, with NOOP i/o
scheduler, tickless kernel. 1 - 1.5GiB of ram each.
- 1x HVM 32bit Windows XP DomU, without any paravirt driver, 512MiB RAM
- I use logical volumes as storage space for DomU's, the linux ones
also have 0.5GiB of swap space (unused, no DomU is swapping)
- all the linux DomU are on ext3 (noatime), and all DomU are single-
cpu (just one vcpu each)
- network is bridged (one lan and one wan interface on the physical
system and the same for each domU), no jumbo frames
Usually load on the system is very low. But when there is some I/O
related load (I can easily trigger it by rsync'ing lots of stuff
between domU's or from a different system to one of the domU or to the
dom0) load gets very high and I often see domU's spending all their
cpu time in "wait" [for I/O] state. When that happens, load on
Domain-0 gets high (jumps from <1 to >5) and loads on DomU's get high
too probably because of processes waiting for I/O to happen. Sometimes
iostat will even show exactly 0.00 tps on all the dm-X devices (domU
storage backends) and some activity on the physical devices, like all
domU I/O activity froze up while dom0 is busy flushing caches or doing
some other stuff.
vmstat in Dom0 shows one or two cores (25% or 50% cpu time) busy in
'iowait' state, and context switches go in the thousands, but not in
the hundreths thousands that http://wiki.xensource.com/xenwiki/KnownIssues
I tried pinning cpus: Domain-0 had its four VCPUs pinned to CPUs 0 and
1, some domU's pinned to CPU 2, and some domU's pinned to CPU 3. As
far as I can tell it did not do any difference.
I also (briefly) tested with all linux DomU's running with the CFQ
scheduler, while it didn't seem to make any difference it also was too
short of a test to trust it much.
What's worse, sometimes I get qemu-dm processes (for the HVM domU) in
zombie state. It also happened that the HVM domU crashed and I wasn't
able to restart it: I got the hotplug scripts not working error from
xm create, and looking in xenstore-ls I saw instances of the crashed
domU with all its resources (which probably was the cause of the
error?). Had the reboot the whole system to be able to start that
Normally iostat in Domain-0 shows more or less high tps (200~300 under
normal load, even higher if I play around with rsync to artificially
trigger the problems) on the md device where all the DomU reside, and
much less (usually just 10-20% of the previous value) on the two
physical disks sda and sdb that compose the mirror. I guess I see less
tps because the scheduler/elevator in Dom-0 is doing its job.
I don't know if the load problems and the HVM problem are linked or
not, but I also don't know where to look to solve any one of them.
Any help would be appreciated, thank you very much. Also, what are
ideal/recommended settings in dom0 and domU regarding i/o schedulers
and tickless or not?
Is there any reason to leave the hypervisor some extra free ram or it
is ok to just let xend shrink dom0 when needed and leave free just the
minimum? If I sum up memory (currently) used by domains, I get
14146MiB. xm info says 14335MiB total_memory and 10MiB free_memory.
Xen-users mailing list
|<Prev in Thread]
||[Next in Thread>|
- [Xen-users] disk I/O problems under load? (Xen-3.4.1/x86_64),
Luca Lesinigo <=