|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] IO speed limited by size of IO request (for RBD driver)
On 8/05/2013 6:33 PM, Roger Pau Monné wrote: On 08/05/13 10:20, Steven Haigh wrote:On 30/04/2013 8:07 PM, Felipe Franciosi wrote:I noticed you copied your results from "dd", but I didn't see any conclusions drawn from experiment. Did I understand it wrong or now you have comparable performance on dom0 and domU when using DIRECT? domU: # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 25.4705 s, 84.3 MB/s dom0: # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 24.8914 s, 86.3 MB/s I think that if the performance differs when NOT using DIRECT, the issue must be related to the way your guest is flushing the cache. This must be generating a workload that doesn't perform well on Xen's PV protocol.Just wondering if there is any further input on this... While DIRECT writes are as good as can be expected, NON-DIRECT writes in certain cases (specifically with a mdadm raid in the Dom0) are affected by about a 50% loss in throughput... The hard part is that this is the default mode of writing!As another test with indirect descriptors, could you change xen_blkif_max_segments in xen-blkfront.c to 128 (it is 32 by default), recompile the DomU kernel and see if that helps? Ok, here we go.... compiled as 3.8.0-2 with the above change. 3.8.0-2 is running on both the Dom0 and DomU.
# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 22.1703 s, 96.9 MB/s
avg-cpu: %user %nice %system %iowait %steal %idle
0.34 0.00 17.10 0.00 0.23 82.33
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdd 980.97 11936.47 53.11 429.78 4.00 48.77
223.81 12.75 26.10 2.11 101.79
sdc 872.71 11957.87 45.98 435.67 3.55 49.30
224.71 13.77 28.43 2.11 101.49
sde 949.26 11981.88 51.30 429.33 3.91 48.90
225.03 21.29 43.91 2.27 109.08
sdf 915.52 11968.52 48.58 428.88 3.73 48.92
225.84 21.44 44.68 2.27 108.56
md2 0.00 0.00 0.00 1155.61 0.00 97.51
172.80 0.00 0.00 0.00 0.00
# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 25.3708 s, 84.6 MB/s
avg-cpu: %user %nice %system %iowait %steal %idle
0.11 0.00 13.92 0.00 0.22 85.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdd 0.00 13986.08 0.00 263.20 0.00 55.76
433.87 0.43 1.63 1.07 28.27
sdc 202.10 13741.55 6.52 256.57 0.81 54.77
432.65 0.50 1.88 1.25 32.78
sde 47.96 11437.57 1.55 261.77 0.19 45.79
357.63 0.80 3.02 1.85 48.60
sdf 2233.37 11756.13 71.93 191.38 8.99 46.80
433.90 1.49 5.66 3.27 86.15
md2 0.00 0.00 0.00 731.93 0.00 91.49
256.00 0.00 0.00 0.00 0.00
Now this is pretty much exactly what I would expect the system to do.... ~96MB/sec buffered, and 85MB/sec direct. So - it turns out that xen_blkif_max_segments at 32 is a killer in the DomU. Now it makes me wonder what we can do about this in kernels that don't have your series of patches against it? And also about the backend stuff in 3.8.x etc? -- Steven Haigh Email: netwiz@xxxxxxxxx Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |