Simon Bettison wrote:
> Hi All,
>
> This is my first time on this (or in fact any as I recall!) list so be
> gentle :)
>
> I am seeing some peculiar load behaviour with a gentoo linux domu which
> is acting as mailserver. The load average starts out fine but then grows
> over the course of several hours topping out at around 120 (one hundred
> and twenty!)
>
> The strange thing is the machine seems responsive, imap connections
> working ok, mail being delivered, snappy shell access etc
>
> This makes me suspicious that the load average is more a reflection of
> the vm load on the cpu slice its being given (ie its only getting say
> 0.01 of the cpu time, and would be running at say 0.11 if it was not
> virtualised) maybe I'm way off though!
>
> The setup is not overly complex:
>
> # xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 1708 1 r-----
> 1682.4
> gentoo-spud 5 1024 1 -b----
> 2792.4
> solaris 2 1024 1 -b----
> 1757.4
>
> its a gentoo-linux dom0, its running on amd-vt hardware, do both domUs
> are unmodified guests...
>
> xen # cat /etc/xen/gentoo-spud
> name = "gentoo-spud";
> memory = 1024;
> disk = [ "file:/var/xen/gentoo/spud/domU,xvda,w" ];
> kernel = "/var/xen/gentoo/spud/vmlinuz";
> root = "/dev/xvda ro";
> vif = [ "mac=00:16:3E:00:00:12" ];
>
> xen # cat /etc/xen/solaris.py
> name = 'solaris'
> memory = '1024'
> disk = [ 'file:/var/xen/solaris/domU,0,w' , 'phy:/dev/sdb,1,w' ,
> 'phy:/dev/sdc,2,w' ,'phy:/dev/sdd,3,w' ]
> vif = [ 'mac=00:16:3E:00:00:11' ]
>
> The solaris vm is providing zfs on the 3 physical devices ie
>
>
> solaris # zpool list
> NAME SIZE USED AVAIL CAP HEALTH ALTROOT
> tank 832G 200G 632G 24% ONLINE -
>
> solaris # zpool status
> pool: tank
> state: ONLINE
> scrub: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> tank ONLINE 0 0 0
> raidz1 ONLINE 0 0 0
> c0d1 ONLINE 0 0 0
> c0d2 ONLINE 0 0 0
> c0d3 ONLINE 0 0 0
>
> solaris # zfs list
> NAME USED AVAIL REFER MOUNTPOINT
> tank 133G 412G 24.0K /tank
> tank/mail 1.32G 412G 1.32G /export/zfs/mail
>
> the tank/mail share is exported over nfs:
>
> solaris # zfs get sharenfs tank/mail
> NAME PROPERTY VALUE SOURCE
> tank/mail sharenfs rw,anon=0 local
>
> the linux box is running:
> postfix
> courier-imapd-ssl
> courier-authlib
> amavisd
> spamassassin
> clamav
>
> mail is routed through postfix on port 25 to amavis on port 10024 which
> takes care of spam and av then back into postfix on 10025 for local delivery
>
> all domains/users are virtual on postfix with delivery to
> /home/vmail/<user@domain>/
>
> /home/vmail is a symbolic link to /mnt/zfs/mail
> /mnt/zfs/mail is nfs mounted share from the solaris vm
>
> through a process of elimination I found that restarting
> courier-imapd-ssl would cause the loadavg to head back towards zero,
> where it would stay for short while but then eventually start to
> escalate again.
>
> I can't find any explanation....
>
> # vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 0 0 0 495912 14488 185348 0 0 34 64 399 296 2 1
> 96 0
> 0 0 0 494296 14488 185352 0 0 0 0 269 97 5 1
> 93 0
> 0 0 0 494296 14488 185352 0 0 0 0 207 82 0 0
> 99 0
> 0 0 0 494288 14488 185400 0 0 0 0 131 64 0 0
> 99 0
> 0 0 0 494288 14488 185452 0 0 0 0 128 60 0 0
> 100 0
> 0 0 0 494304 14488 185472 0 0 0 8 130 74 0 0
> 99 0
> 0 0 0 494304 14488 185472 0 0 0 0 144 67 0 0
> 100 0
> 0 0 0 494368 14488 185512 0 0 0 0 127 62 0 0
> 99 0
> 0 0 0 494260 14488 185512 0 0 0 0 141 73 0 0
> 100 0
> 0 0 0 494252 14488 185556 0 0 0 0 143 65 0 0
> 99 0
> 0 0 0 494252 14520 185576 0 0 0 32 116 68 0 0
> 99 0
> 0 0 0 494300 14520 185596 0 0 0 0 127 69 0 0
> 100 0
> 0 0 0 494192 14520 185596 0 0 0 0 141 68 0 0
> 99 0
> 0 0 0 494184 14520 185640 0 0 0 0 153 60 0 0
> 99 0
> 0 0 0 494184 14520 185640 0 0 0 0 122 59 0 0
> 100 0
> 0 0 0 494216 14520 185680 0 0 0 0 124 72 0 0
> 99 0
> 0 0 0 494108 14520 185680 0 0 0 0 123 67 0 0
> 100 0
> 0 0 0 494092 14520 185720 0 0 0 0 117 63 0 0
> 99 0
> 0 0 0 494092 14520 185720 0 0 0 0 119 64 0 0
> 100 0
> 0 0 0 494124 14520 185760 0 0 0 0 123 66 0 0
> 99 0
> 0 0 0 494124 14520 185760 0 0 0 0 125 70 0 0
> 99 0
> 0 0 0 494016 14520 185800 0 0 0 0 118 60 0 0
> 100 0
> 0 0 0 494016 14520 185800 0 0 0 0 93 42 0 0
> 97 0
> 0 0 0 494032 14520 185820 0 0 0 0 25 18 0 0
> 99 0
> 0 0 0 494032 14520 185820 0 0 0 0 11 12 0 0
> 100 0
> 0 0 0 494032 14520 185828 0 0 0 0 89 55 0 0
> 100 0
> 0 0 0 494032 14520 185828 0 0 0 0 138 68 0 1
> 98 0
> 0 0 0 494040 14520 185872 0 0 0 0 118 66 0 0
> 99 0
> 0 0 0 493932 14520 185872 0 0 0 0 122 63 0 0
> 99 0
> 0 0 0 493916 14520 185912 0 0 0 0 129 62 0 0
> 100 0
> 0 0 0 493916 14552 185932 0 0 0 44 126 76 0 0
> 99 0
> 0 0 0 493916 14552 185956 0 0 0 0 123 65 0 0
> 100 0
> 0 0 0 493808 14552 185956 0 0 0 0 119 58 0 0
> 99 0
> 0 0 0 493800 14552 185996 0 0 0 0 132 71 0 0
> 100 0
> 0 0 0 493800 14552 185996 0 0 0 0 120 62 0 0
> 99 0
> 0 0 0 493800 14552 186040 0 0 0 36 131 73 0 0
> 99 0
> 0 0 0 493692 14552 186040 0 0 0 0 133 68 0 0
> 100 0
> 0 0 0 493676 14552 186080 0 0 0 0 130 66 0 0
> 99 0
> 0 0 0 493676 14552 186080 0 0 0 0 125 63 0 0
> 99 0
> 0 0 0 493684 14552 186124 0 0 0 0 139 79 0 0
> 100 0
> 0 0 0 493684 14552 186124 0 0 0 0 126 68 0 0
> 99 0
> 0 0 0 493584 14552 186164 0 0 0 0 125 62 0 0
> 99 0
> 0 0 0 493584 14552 186164 0 0 0 0 127 60 0 0
> 100 0
> 0 0 0 493584 14552 186204 0 0 0 0 130 80 0 0
> 99 0
> 0 0 0 493584 14552 186204 0 0 0 0 132 69 0 0
> 99 0
> 0 0 0 493460 14564 186248 0 0 0 12 119 74 0 0
> 100 0
> 0 0 0 493460 14564 186248 0 0 0 0 129 68 0 0
> 99 0
> 0 0 0 493452 14564 186288 0 0 0 0 145 75 0 0
> 100 0
>
> thought maybe it was nfs...?
>
> # iostat -n 1
> Linux 2.6.21-xen (spud) 05/28/08
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 10363.60 13.49 0.00
> 0.00 11.30 13.49
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 24.18 24.19 0.00
> 0.00 0.00 18.67
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 0.00 0.05 0.00
> 0.00 0.00 5.56
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 1.98 0.06 0.00
> 0.00 0.00 0.05
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 4137.37 0.06 0.00
> 0.00 0.00 0.07
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 12051.17 4.16 0.00
> 0.00 0.00 4.16
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 6151.21 7.25 0.00
> 0.00 0.00 7.26
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 15.48 15.49 0.00
> 0.00 0.00 15.49
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 26.00 26.00 0.00
> 0.00 0.00 26.00
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 26.98 27.00 0.00
> 0.00 0.00 21.00
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 26.79 26.79 0.00
> 0.00 0.00 29.09
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 20.99 21.00 0.00
> 0.00 0.00 18.77
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 21.20 21.24 0.00
> 0.00 0.00 27.23
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 0.00 0.05 0.00
> 0.00 0.00 0.05
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 1.98 0.05 0.00
> 0.00 0.00 0.05
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 0.00 0.05 0.00
> 0.00 0.00 0.05
>
> Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s
> wBlk_dir/s rBlk_svr/s wBlk_svr/s
> swanney:/export/zfs/mail 2.75 2.79 0.00
> 0.00 0.00 2.79
>
> the machine is hardly begin touched - only two mailboxes on it, very
> litle mail passing through.
>
> any pointers would be really appreciated
>
> Simon Bettison
>
I'm not sure what exactly the issue was, but i think it might have
something to do with courier-imap + thousands of e-mails (~20k total
with some maildirs having 5k in) + NFS
installed dovecot insted and this seems to be working fine.
Simon
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home