This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-users] LVM Read/Write speed <10% drive's normal speed: SOLVED

To: "Olivier B." <xen.list@xxxxxxxxx>, "Bryn M. Reeves" <bmr@xxxxxxxxxx>, Robert Dunkley <Robert@xxxxxxxxx>, Jeff Sturm <jeff.sturm@xxxxxxxxxx>, "Fajar A. Nugraha" <fajar@xxxxxxxxx>
Subject: Re: [Xen-users] LVM Read/Write speed <10% drive's normal speed: SOLVED
From: Stephen Hamer <Stephen.Hamer@xxxxxxxxxx>
Date: Thu, 17 Sep 2009 18:05:52 +0000
Cc: "xen-users@xxxxxxxxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxxxxxxxx>, LVM general discussion and development <linux-lvm@xxxxxxxxxx>
Delivery-date: Thu, 17 Sep 2009 11:06:55 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4AB1E92F.4010109@xxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Organization: JHUAPL
References: <1253137773.5136.357.camel@hamerst1-ll1> <4AB1E92F.4010109@xxxxxxxxx>
Reply-to: Stephen.Hamer@xxxxxxxxxx
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
WOW. Thank you all for the overwhelming response!!! I'm going to answer you all in order of who replied first.
(NOTE: the solution is in the 2nd section. Thanks Oliver!)

On Thu, 2009-09-17 at 03:41 -0400, Robert Dunkley wrote:
>Hi Stephen,
>Can you please post your DomU config file to the list.

Sure, without any of the comments, here it is (without the true names or IPs, of course):

kernel      = '/boot/vmlinuz-2.6.26-2-xen-686'
ramdisk     = '/boot/initrd.img-2.6.26-2-xen-686'
memory      = '1024'
root        = '/dev/hda2 ro'
disk        = [ 'phy:/dev/xenvg/testVM1-swap,hda1,w',
                  'phy:/dev/xenvg/testVM1-root,hda2,w', ]

name        = 'testVM1'

vif = [ 'ip=' ]

on_reboot   = 'restart'
on_crash    = 'restart'

# Needs to be inserted to allow console to attach to VM
extra="console=hvc0 xencons=tty"

This was generated by xen-tools (after a little modification to xen-tools xm.tmpl). I like how easy xen-tools is to expand upon. For example, I used 'xen-update-image' as a model to create my own 'xen-image-cmd' which can pass a single command to multiple offline domUs. Works just like 'xen-update-image', except for the fact that you can pass ANY command that the domU can execute AND you can force the output to be redirected to a file INSIDE the chrooted image. I was sort of proud of myself for that scripting voodoo. =D

On Thu, 2009-09-17 at 03:45 -0400, Olivier B. wrote:

>what mount options do you use on the DomU ?
>Under Debian there is xen-tools example with the "sync" mount options : 
>if you use it, writes will be really slow.


THIS is where the money is. Once I removed the 'sync' option from the fstab for my root partition, the I/O speed sailed. Makes sense, really. This got the VM's performance to actually BEAT the old bare metal machine that we were running this server on before.
Thanks for your help!

On Thu, 2009-09-17 at 06:21 -0400, Bryn M. Reeves wrote:
What is the configuration of the guest's storage? I.e. what type of virt
driver are you using to expose the dom0 storage to the domUs? This can
have a massive impact on the performance levels you can achieve.

You should also test the performance of these underlying devices in the
domUs - i.e. benchmark read (and if possible write) performance of
the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
other type of virtual disks you are using).


I'm not PRECISELY sure what you mean here, but I think you're referring to the xen driver used to pass the storage in as a block dev. to the domU. If that is what you mean, I used the 'phy:' command. The performance of the /dev/hda* inside the domU is what I was mentioning trying to benchmark simplistically with 'dd'.
- A 'dd' command in my dom0 on the true physical /dev/sda1 got 100 MB/s.
- Before I turned off 'sync', a 'dd' command on /dev/hda2 inside my domU (physically a 10GB LVM on the physical device /dev/sda5) got 3 MB/s.
I noted later that running a DD command directly to another LVM on that same VG as the others got comparable performance to the 100 MB/s. That was my first indicator that the domUs were doing something nasty.
- After I turned off 'sync', a 'dd' command on that same /dev/hda2 inside the same domU got 100 MB/s!

On Thu, 2009-09-17 at 10:59 -0400, Jeff Sturm wrote:
Your domU's--are they paravirt, or HVM?
I believe they're paravirtualized, unless I'm doing something silly. They're running off the modified kernel, and I'm not doing any hardware forwarding magic. See config file at the top.

On Thu, 2009-09-17 at 11:16 -0400, Fajar A. Nugraha wrote:
I suggest you try using official xen.org's 2.6.18 kernel, or other
vendor-supported 2.6.18 kernel-xen (like RHEL5's kernel-xen, or even
Etch's kernel) for both domU and dom0 and see if the problem persists.
I've noticed a couple of quirks in the startup sequence of my domUs, I figured it had something interesting to do with the kernel. I'll look into using one of those kernels. Thanks!
Xen-users mailing list