|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] IO speed limited by size of IO request (for RBD driver)
On 27/04/13 10:35, Steven Haigh wrote: > On 27/04/2013 5:51 PM, Steven Haigh wrote: >> On 27/04/2013 5:06 PM, Roger Pau Monné wrote: >>> On 27/04/13 03:57, Steven Haigh wrote: >>>> On 27/04/2013 12:16 AM, Steven Haigh wrote: >>>>> On 27/04/2013 12:06 AM, Roger Pau Monné wrote: >>>>>> On 23/04/13 21:05, Steven Haigh wrote: >>>>>>> Sorry - resending this to Felipe as well - as I started talking to >>>>>>> him >>>>>>> directly previously. >>>>>>> >>>>>>> Felipe, to bring you up to date, I've copied over the blkback >>>>>>> files from >>>>>>> Rogers indirect kernel over the vanilla 3.8.8 kernel files, built and >>>>>>> tested. Results below: >>>>>>> >>>>> >>>>> Bringing this into context in a nutshell - results showed about 5MB/sec >>>>> improvement when using buffered disk access - totalling ~57MB/sec write >>>>> speed vs ~98MB/sec when using the oflag=direct flag to dd. >>>>> >>>>> When talking about back porting a few indirect patches to mainline >>>>> blkback (3.8.8 atm): >>>>>>> On 24/04/2013 4:13 AM, Roger Pau Monné wrote: >>>>>>>> I think it requires a non-trivial amount of work, what you could do >>>>>>>> as a >>>>>>>> test is directly replace the affected files with the ones in my >>>>>>>> tree, it >>>>>>>> is not optimal, but I don't think it's going to cause problems, >>>>>>>> and you >>>>>>>> could at least see if indirect descriptors solve your problem. >>>>>>> >>>>>>> Ok, I copied across those files, built, packaged and installed >>>>>>> them on >>>>>>> my Dom0. Good news is that its a little quicker, bad news is not by >>>>>>> much. >>>>>> >>>>>> Could you try increasing xen_blkif_max_segments variable in >>>>>> xen-blkfront.c to 64 or 128? It is set to 32 by default. You will only >>>>>> need to recompile the DomU kernel after this, the Dom0 is able to >>>>>> support up to 256 indirect segments. >>>>> >>>>> I'll have to look at this. All DomU's are Scientific Linux 6.4 >>>>> systems - >>>>> so essentially RHEL6.4 and so on. I haven't built a RH kernel as yet - >>>>> so I'll have to look at what is involved. It might be as simple as >>>>> rebuilding a normal SRPM. >>>> >>>> Ok, I've had a look at the RH xen-blkfront.c - and I can't see any >>>> definition of xen_blkif_max_segments - or anything close. I've attached >>>> the version used in the EL6 kernel from the kernel-2.6.32-358.6.1.el6 >>>> srpm. >>>> >>>> Any ideas on where to go from here? >>> >>> I thought you were using the 3.8.x kernel inside the DomU also, if you >>> are not using it, then it's normal that there's no speed difference, you >>> have a Dom0 kernel that supports indirect descriptors, but your DomU >>> doesn't. You must use a kernel that supports indirect descriptors in >>> both Dom0 and DomU in order to make use of this feature. >> >> Ahhh - sorry - I should have been clearer - The Dom0 is kernel 3.8.x >> (3.8.8 right now) - however the DomUs are all stock EL6 kernels. >> >> Hmmmm - I believe the kernel I build for Dom0 *should* work as a DomU. >> I'll do some more experimentation and see if I can get it working >> properly as a DomU kernel. > > Ok - now for testing the basics. > > Same tests using vanilla 3.8.8 kernel: > # dd if=/dev/zero of=output.zero bs=1M count=2048 > 2048+0 records in > 2048+0 records out > 2147483648 bytes (2.1 GB) copied, 37.1206 s, 57.9 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 2.65 0.00 0.22 97.13 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdf 395.81 2201.32 27.59 156.95 1.65 9.21 > 120.60 1.13 6.12 1.19 22.05 > sde 404.86 2208.83 28.04 157.40 1.69 9.24 > 120.77 1.32 7.15 1.31 24.24 > sdc 435.54 2174.83 30.68 155.63 1.82 9.10 > 120.09 0.97 5.20 1.11 20.64 > sdd 388.96 2177.26 26.71 155.41 1.62 9.11 > 120.74 1.10 6.01 1.30 23.60 > md2 0.00 0.00 0.00 537.31 0.00 17.59 > 67.05 0.00 0.00 0.00 0.00 > > # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct > 2048+0 records in > 2048+0 records out > 2147483648 bytes (2.1 GB) copied, 25.3928 s, 84.6 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.22 0.00 15.74 0.00 0.22 83.81 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdf 336.81 12659.65 10.86 232.59 1.36 50.36 > 435.06 1.00 4.09 2.51 61.06 > sde 1684.04 11000.22 54.32 189.14 6.79 43.71 > 424.80 1.45 5.95 3.50 85.28 > sdc 144.35 11177.61 4.66 238.80 0.58 44.60 > 380.04 0.41 1.70 1.07 26.08 > sdd 20.62 12876.50 0.67 242.79 0.08 51.25 > 431.80 0.45 1.84 1.15 27.92 > md2 0.00 0.00 0.00 2680.71 0.00 86.47 > 66.06 0.00 0.00 0.00 0.00 > > Installed and rebooted into the patched version I build by just copying > the blkback files across to the 3.8.8 tree and building: Did you also copy xen-blkfront? > > # dd if=/dev/zero of=output.zero bs=1M count=2048 > 2048+0 records in > 2048+0 records out > 2147483648 bytes (2.1 GB) copied, 45.2376 s, 47.5 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 1.35 0.00 0.45 98.19 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdd 1340.80 5806.80 158.20 674.40 5.83 25.27 > 76.51 6.00 7.16 0.80 66.90 > sde 1334.60 5894.00 160.80 686.40 5.86 25.71 > 76.32 6.87 8.11 0.87 73.52 > sdc 1330.80 5858.20 158.00 682.40 5.86 25.60 > 76.67 5.71 6.81 0.77 64.84 > sdf 1341.00 5848.80 157.00 681.20 5.83 25.49 > 76.53 6.23 7.38 0.85 70.92 > md2 0.00 0.00 0.00 1431.40 0.00 46.83 > 67.01 0.00 0.00 0.00 0.00 > > # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct > 2048+0 records in > 2048+0 records out > 2147483648 bytes (2.1 GB) copied, 38.9052 s, 55.2 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 5.27 0.00 0.32 94.41 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdd 493.20 8481.60 36.80 335.80 2.07 34.45 > 200.73 1.14 3.07 0.97 36.32 > sde 1371.60 7380.20 83.80 304.80 5.66 30.08 > 188.34 2.20 5.65 1.94 75.38 > sdc 540.20 7556.80 56.00 326.20 2.33 30.80 > 177.52 1.49 3.90 1.26 48.02 > sdf 734.20 8286.60 64.40 326.20 3.12 33.67 > 192.92 1.66 4.24 1.45 56.66 > md2 0.00 0.00 0.00 1835.20 0.00 59.20 > 66.06 0.00 0.00 0.00 0.00 > > That is with the same kernel running on both Dom0 and DomU. > > In the dmesg of the DomU, I see the following: > blkfront: xvdb: flush diskcache: enabled using persistent grants It seems you are missing some pieces, you should see something like: blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |