|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] IO speed limited by size of IO request (for RBD driver)
On 27/04/2013 5:51 PM, Steven Haigh wrote: On 27/04/2013 5:06 PM, Roger Pau Monné wrote:On 27/04/13 03:57, Steven Haigh wrote:On 27/04/2013 12:16 AM, Steven Haigh wrote:On 27/04/2013 12:06 AM, Roger Pau Monné wrote:On 23/04/13 21:05, Steven Haigh wrote:Sorry - resending this to Felipe as well - as I started talking to him directly previously. Felipe, to bring you up to date, I've copied over the blkback files from Rogers indirect kernel over the vanilla 3.8.8 kernel files, built and tested. Results below:Bringing this into context in a nutshell - results showed about 5MB/sec improvement when using buffered disk access - totalling ~57MB/sec write speed vs ~98MB/sec when using the oflag=direct flag to dd. When talking about back porting a few indirect patches to mainline blkback (3.8.8 atm):On 24/04/2013 4:13 AM, Roger Pau Monné wrote:I think it requires a non-trivial amount of work, what you could do as a test is directly replace the affected files with the ones in my tree, it is not optimal, but I don't think it's going to cause problems, and you could at least see if indirect descriptors solve your problem.Ok, I copied across those files, built, packaged and installed them on my Dom0. Good news is that its a little quicker, bad news is not by much.Could you try increasing xen_blkif_max_segments variable in xen-blkfront.c to 64 or 128? It is set to 32 by default. You will only need to recompile the DomU kernel after this, the Dom0 is able to support up to 256 indirect segments.I'll have to look at this. All DomU's are Scientific Linux 6.4 systems - so essentially RHEL6.4 and so on. I haven't built a RH kernel as yet - so I'll have to look at what is involved. It might be as simple as rebuilding a normal SRPM.Ok, I've had a look at the RH xen-blkfront.c - and I can't see any definition of xen_blkif_max_segments - or anything close. I've attached the version used in the EL6 kernel from the kernel-2.6.32-358.6.1.el6 srpm. Any ideas on where to go from here?I thought you were using the 3.8.x kernel inside the DomU also, if you are not using it, then it's normal that there's no speed difference, you have a Dom0 kernel that supports indirect descriptors, but your DomU doesn't. You must use a kernel that supports indirect descriptors in both Dom0 and DomU in order to make use of this feature.Ahhh - sorry - I should have been clearer - The Dom0 is kernel 3.8.x (3.8.8 right now) - however the DomUs are all stock EL6 kernels. Hmmmm - I believe the kernel I build for Dom0 *should* work as a DomU. I'll do some more experimentation and see if I can get it working properly as a DomU kernel.
Ok - now for testing the basics.
Same tests using vanilla 3.8.8 kernel:
# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 37.1206 s, 57.9 MB/s
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 2.65 0.00 0.22 97.13
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdf 395.81 2201.32 27.59 156.95 1.65 9.21
120.60 1.13 6.12 1.19 22.05
sde 404.86 2208.83 28.04 157.40 1.69 9.24
120.77 1.32 7.15 1.31 24.24
sdc 435.54 2174.83 30.68 155.63 1.82 9.10
120.09 0.97 5.20 1.11 20.64
sdd 388.96 2177.26 26.71 155.41 1.62 9.11
120.74 1.10 6.01 1.30 23.60
md2 0.00 0.00 0.00 537.31 0.00 17.59
67.05 0.00 0.00 0.00 0.00
# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 25.3928 s, 84.6 MB/s
avg-cpu: %user %nice %system %iowait %steal %idle
0.22 0.00 15.74 0.00 0.22 83.81
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdf 336.81 12659.65 10.86 232.59 1.36 50.36
435.06 1.00 4.09 2.51 61.06
sde 1684.04 11000.22 54.32 189.14 6.79 43.71
424.80 1.45 5.95 3.50 85.28
sdc 144.35 11177.61 4.66 238.80 0.58 44.60
380.04 0.41 1.70 1.07 26.08
sdd 20.62 12876.50 0.67 242.79 0.08 51.25
431.80 0.45 1.84 1.15 27.92
md2 0.00 0.00 0.00 2680.71 0.00 86.47
66.06 0.00 0.00 0.00 0.00
Installed and rebooted into the patched version I build by just copying the blkback files across to the 3.8.8 tree and building:
# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 45.2376 s, 47.5 MB/s
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 1.35 0.00 0.45 98.19
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdd 1340.80 5806.80 158.20 674.40 5.83 25.27
76.51 6.00 7.16 0.80 66.90
sde 1334.60 5894.00 160.80 686.40 5.86 25.71
76.32 6.87 8.11 0.87 73.52
sdc 1330.80 5858.20 158.00 682.40 5.86 25.60
76.67 5.71 6.81 0.77 64.84
sdf 1341.00 5848.80 157.00 681.20 5.83 25.49
76.53 6.23 7.38 0.85 70.92
md2 0.00 0.00 0.00 1431.40 0.00 46.83
67.01 0.00 0.00 0.00 0.00
# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 38.9052 s, 55.2 MB/s
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 5.27 0.00 0.32 94.41
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdd 493.20 8481.60 36.80 335.80 2.07 34.45
200.73 1.14 3.07 0.97 36.32
sde 1371.60 7380.20 83.80 304.80 5.66 30.08
188.34 2.20 5.65 1.94 75.38
sdc 540.20 7556.80 56.00 326.20 2.33 30.80
177.52 1.49 3.90 1.26 48.02
sdf 734.20 8286.60 64.40 326.20 3.12 33.67
192.92 1.66 4.24 1.45 56.66
md2 0.00 0.00 0.00 1835.20 0.00 59.20
66.06 0.00 0.00 0.00 0.00
That is with the same kernel running on both Dom0 and DomU. In the dmesg of the DomU, I see the following: blkfront: xvdb: flush diskcache: enabled using persistent grants -- Steven Haigh Email: netwiz@xxxxxxxxx Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |