WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] IOwaits over NFS

To: "Ribeiro Emerson Gomes" <Emerson.Ribeiro@xxxxxxxxxx>, <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: Re: [Xen-users] IOwaits over NFS
From: "Geoff Galitz" <geoff@xxxxxxxxxx>
Date: Tue, 14 Sep 2010 17:07:57 +0200
Cc:
Delivery-date: Tue, 14 Sep 2010 08:10:50 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
Importance: Normal
In-reply-to: <8CBB53B8098D4544A6FF75CFFAD77DF2CA3C72@xxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <8CBB53B8098D4544A6FF75CFFAD77DF2CA3C72@xxxxxxxxxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
 
I neglected to mention that the problem is specifically for NFS mounts within the DomUs. 
 
Regardless, I'll make a point of continuing our tuning using dd.
 
Thanks.
 
 

Sent: Tuesday, September 14, 2010 4:42 PM
Subject: RE: [Xen-users] IOwaits over NFS

rsize and wsize are the most important ones to reduce your iowait. Those can dramatically change performance.

The theory is that the best performance is reached when you match the geometry of the actual disks to the rsize/wsize nfs parameter.

Use dd to emulate different block sizes and check the best performance for your environment.

 

Emerson

 

 

-----Original Message-----
From: Geoff Galitz [mailto:geoff@xxxxxxxxxx]
Sent: Tuesday, September 14, 2010 11:33 AM
To: Ribeiro Emerson Gomes; xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] IOwaits over NFS

 

 

 

We`ve used various NFS parameters in our testing to resolve this.  Here are three different  examples of what we have used on a single machine, with no significant improvement:

 

1) defaults,auto,tcp,bg,hard,intr,noatime 0 0

2) auto,noatime,rsize=32768,wsize=32768,bg,nfsvers=3,intr,tcp,retrans=0 0 0

3) noatime,bg,intr,nolock,tcp      0 0

 

 

Thanks.

 

 

 

Sent: Tuesday, September 14, 2010 4:20 PM

Subject: RE: [Xen-users] IOwaits over NFS

 

Please inform the parameters you use in fstab when you mount your NFS share.

Something like this:

<server>:/<share> <mountpoint> nfs rsize=8192, wsize=8192, timeo=14, intr, tcp, bg, rw, bg, hard, nointr, rsize=32768, wsize=32768, tcp, actimeo=0, vers=3,timeo=600

 

 

Emerson

 

 

-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Geoff Galitz
Sent:
Tuesday, September 14, 2010 11:04 AM
To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] IOwaits over NFS

 

Hello.

We have a number of Xen 3.4.2. boxes which have constant iowaits at around
10% with spikes up to 100% when accessing data over NFS.  We have been
unable to nail down the issue.  Any advice?


System info:


release                : 2.6.18-194.3.1.el5xen
version                : #1 SMP Thu May 13
13:49:53 EDT 2010
machine                : x86_64
nr_cpus                : 16
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2261
hw_caps                :
bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 32758
free_memory            : 3974
node_to_cpu            : node0:0-15
node_to_memory         : node0:3974
xen_major              : 3
xen_minor              : 4
xen_extra              : .2
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
cc_compile_by          : root
cc_compile_domain      : gitco.tld
cc_compile_date        : Wed Nov 11 21:16:28 CET 2009
xend_config_format     : 4


And lsmod from a VM:

Module                  Size  Used by
nfsd                  284041  17
exportfs               38849  1 nfsd
auth_rpcgss            81889  1 nfsd
ipv6                  424737  24
xfrm_nalgo             43333  1 ipv6
crypto_api             42945  1 xfrm_nalgo
i2c_dev                43849  0
i2c_core               56129  1 i2c_dev
nfs                   289549  1
lockd                  99185  3 nfsd,nfs
fscache                52385  1 nfs
nfs_acl                36673  2 nfsd,nfs
sunrpc                197513  18 nfsd,auth_rpcgss,nfs,lockd,nfs_acl
iptable_filter         36161  1
ip_tables              55329  1 iptable_filter
x_tables               50377  1 ip_tables
dm_mirror              54345  0
dm_multipath           55385  0
scsi_dh                41665  1 dm_multipath
scsi_mod              197337  1 scsi_dh
parport_pc             62313  0
lp                     47121  0
parport                73293  2 parport_pc,lp
pcspkr                 36289  0
xennet                 61129  0 [permanent]
dm_raid45              98897  0
dm_message             36161  1 dm_raid45
dm_region_hash         46401  1 dm_raid45
dm_log                 44993  3 dm_mirror,dm_raid45,dm_region_hash
dm_mod                100753  4 dm_mirror,dm_multipath,dm_raid45,dm_log
dm_mem_cache           39617  1 dm_raid45
ext3                  168529  1
jbd                    94257  1 ext3
uhci_hcd               57561  0
ohci_hcd               56053  0
ehci_hcd               65869  0
xenblk                 50729  1

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
<Prev in Thread] Current Thread [Next in Thread>