Xen project Mailing List

Re: [Xen-devel] 4.2.1: Poor write performance for DomU.

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Date: Thu, 05 Sep 2013 18:28:25 +1000

Cc: roger.pau@xxxxxxxxxx, xen-devel@xxxxxxxxxxxxx

Delivery-date: Thu, 05 Sep 2013 08:29:20 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

Openpgp: id=7A7D31DC

On 21/08/13 02:48, Konrad Rzeszutek Wilk wrote: > On Mon, Mar 25, 2013 at 01:21:09PM +1100, Steven Haigh wrote: >> So, based on my tests yesterday, I decided to break the RAID6 and >> pull a drive out of it to test directly on the 2Tb drives in >> question. >> >> The array in question: >> # cat /proc/mdstat >> Personalities : [raid1] [raid6] [raid5] [raid4] >> md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5] >> 3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2 >> [4/4] [UUUU] >> >> # mdadm /dev/md2 --fail /dev/sdf >> mdadm: set /dev/sdf faulty in /dev/md2 >> # mdadm /dev/md2 --remove /dev/sdf >> mdadm: hot removed /dev/sdf from /dev/md2 >> >> So, all tests are to be done on /dev/sdf. >> Model Family: Seagate SV35 >> Device Model: ST2000VX000-9YW164 >> Serial Number: Z1E17C3X >> LU WWN Device Id: 5 000c50 04e1bc6f0 >> Firmware Version: CV13 >> User Capacity: 2,000,398,934,016 bytes [2.00 TB] >> Sector Sizes: 512 bytes logical, 4096 bytes physical >> >> From the Dom0: >> # dd if=/dev/zero of=/dev/sdf bs=1M count=4096 oflag=direct >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) copied, 30.7691 s, 140 MB/s >> >> Create a single partition on the drive, and format it with ext4: >> Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes >> 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 4096 bytes >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes >> Disk identifier: 0x98d8baaf >> >> Device Boot Start End Blocks Id System >> /dev/sdf1 2048 3907029167 1953513560 83 Linux >> >> Command (m for help): w >> >> # mkfs.ext4 -j /dev/sdf1 >> ...... >> Writing inode tables: done >> Creating journal (32768 blocks): done >> Writing superblocks and filesystem accounting information: done >> >> Mount it on the Dom0: >> # mount /dev/sdf1 /mnt/esata/ >> # cd /mnt/esata/ >> # bonnie++ -d . -u 0:0 >> .... >> Version 1.96 ------Sequential Output------ --Sequential >> Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- >> --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec >> %CP /sec %CP >> xenhost.lan.crc. 2G 425 94 133607 24 60544 12 973 95 209114 >> 17 296.4 6 >> Latency 70971us 190ms 221ms 40369us 17657us >> 164ms >> >> So from the Dom0: 133Mb/sec write, 209Mb/sec read. >> >> Now, I'll attach the full disk to a DomU: >> # xm block-attach zeus.vm phy:/dev/sdf xvdc w >> >> And we'll test from the DomU. >> >> # dd if=/dev/zero of=/dev/xvdc bs=1M count=4096 oflag=direct >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) copied, 32.318 s, 133 MB/s >> >> Partition the same as in the Dom0 and create an ext4 filesystem on it: >> >> I notice something interesting here. In the Dom0, the device is seen as: >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 4096 bytes >> I/O size (minimum/optimal): 4096 bytes / 4096 bytes >> >> In the DomU, it is seen as: >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 512 bytes >> I/O size (minimum/optimal): 512 bytes / 512 bytes >> >> Not sure if this could be related - but continuing testing: >> Device Boot Start End Blocks Id System >> /dev/xvdc1 2048 3907029167 1953513560 83 Linux >> >> # mkfs.ext4 -j /dev/xvdc1 >> .... >> Allocating group tables: done >> Writing inode tables: done >> Creating journal (32768 blocks): done >> Writing superblocks and filesystem accounting information: done >> >> # mount /dev/xvdc1 /mnt/esata/ >> # cd /mnt/esata/ >> # bonnie++ -d . -u 0:0 >> .... >> Version 1.96 ------Sequential Output------ --Sequential >> Input- --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- >> --Block-- --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec >> %CP /sec %CP >> zeus.crc.id.au 2G 396 99 116530 23 50451 15 1035 99 176407 >> 23 313.4 9 >> Latency 34615us 130ms 128ms 33316us 74401us >> 130ms >> >> So still... 116Mb/sec write, 176Mb/sec read to the physical device >> from the DomU. More than acceptable. >> >> It leaves me to wonder.... Could there be something in the Dom0 >> seeing the drives as 4096 byte sectors, but the DomU seeing it as >> 512 byte sectors cause an issue? > > There is certain overhead in it. I still have this in my mailbox > so I am not sure whether this issue got ever resolved? I know that the > indirect patches in Xen blkback and xen blkfront are meant to resolve > some of these issues - by being able to carry a bigger payload. > > Did you ever try v3.11 kernel in both dom0 and domU? Thanks. Ok, so I finally got around to building kernel 3.11 RPMs today for testing. I upgraded both the Dom0 and DomU to the same kernel: DomU: # dmesg | grep blkfront blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; Looks good. Transfer tests using bonnie++ as per before: # bonnie -d . -u 0:0 Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP zeus.crc.id.au 2G 603 92 58250 9 62248 14 886 99 295757 30 492.3 13 Latency 27305us 124ms 158ms 34222us 16865us 374ms Version 1.96 ------Sequential Create------ --------Random Create-------- zeus.crc.id.au -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 10048 22 +++++ +++ 17849 29 11109 25 +++++ +++ 18389 31 Latency 17775us 154us 180us 16008us 38us 58us Still seems to be a massive discrepancy between Dom0 and DomU write speeds. Interesting is that sequential block reads are nearly 300MB/sec, yet sequential writes were only ~58MB/sec. -- Steven Haigh Email: netwiz@xxxxxxxxx Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.