Here's my data so far:
I'm just wondering what the block device layer is doing to kill
performance. I'm looking for any buffering directives in the xend
config that I may be able to modify.
iSCSI Performance Profiling
iSCSI target on Heart, Initiator is Dom0 br04-61 as LV
Version 1.94 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
br04-61.adtech2. 8G 110289 29 33781 3 45053 1 1052 9
Latency 2610ms 312ms 191ms 291ms
Version 1.94 ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 627us 157us 235us 506us 183us 282us
1.93c,1.94,br04-61,1,1231849837,8G,,,,110289,29,33781,3,,,450s
iSCSI target on Heart, Initiator on Dom0 br04-61 as LV, bonnie++ on
DomU aenu via xvd
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 47100 3 15069 0 39072 0 3971 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
Local Logical Volume on Dom0 br04-61, bonnie++ on Dom0
Version 1.94 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
br04-61. 8G 39000 7 21419 0 62958 0 158.0 1
Latency 16524ms 1111ms 54552us 565ms
Version 1.94 ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 29249 40 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 22439us 281us 227us 589us 220us 253us
1.93c,1.94,br04-61,1,1231848192,8G,,,,39000,7,21419,0,,,62958,0,158.0,1,16,,,,,29249,40,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,16524ms,1111ms,,54552us,565ms,22439us,281us,227us,589us,220us,253us
Local Logical Volume on Dom0 br04-61, bonnie++ on DomU aenu via xvd
RAM was reduced to 512M on the DomU
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 1G 64860 5 23617 0 58329 0 212.3 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6038 96 +++++ +++ +++++ +++ 6333 99 +++++ +++ 19371 100
aenu,1G,,,64860,5,23617,0,,,58329,0,212.3,0,16,6038,96,+++++,+++,+++++,+++,6333,99,+++++,+++,19371,10
iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++ on br04-61
Version 1.94 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
br04-61.adtech2. 8G 111523 30 34132 3 58284 3 1184 11
Latency 4579ms 250ms 72926us 44558us
Version 1.94 ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 4643us 123us 191us 7013us 153us 195us
1.93c,1.94,br04-61,1,1231852980,8G,,,,111523,30,34132,3,,,58284,3,1184,11,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,4579ms,250ms,,72926us,44558us,4643us,123us,191us,7013us,153us,195us
iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++
on aenu domU as xvd
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 52625 4 15065 0 46256 0 3859 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,52625,4,15065,0,,,46256,0,3859.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
iSCSI target on Heart, Initiator on DomU aenu, bonnie++ on aenu DomU as /dev/sd
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 129048 22 28832 2 49896 0 5174 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,129048,22,28832,2,,,49896,0,5173.6,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
On Tue, Jan 13, 2009 at 3:17 PM, Barry van Someren
<barry@xxxxxxxxxxxxxxxx> wrote:
> Hi there,
>
> I've used Citrix Xenserver 5.0 before this and never experienced this problem.
> I did however store the entire VM on the LUN instead of mounting it to the VM.
>
> Maybe you can trying doing that and seeing if it makes a big difference?
>
> Regards,
>
> Barry
>
> (Failed to send this to the entire list, so sending again, sorry)
>
> On Tue, Jan 13, 2009 at 11:48 PM, Christopher Chen <muffaleta@xxxxxxxxx>
> wrote:
>> Hi there!
>>
>> I've been wrestling with an issue for a little bit now--
>>
>> In my test environment, I have tgtd running on a Centos 5.2 box, with
>> a raid 10 array backing it.
>>
>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland
>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
>>
>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads.
>>
>> I've attached the iSCSI LUN to the DomU as a virtual block device, and
>> I'm seeing 47MB/sec writes, and 39MB/sec reads.
>>
>> I've tried a few things, like running against a local disk, and
>> suprisingly, writes on the DomU are faster than the Dom0--can I assume
>> the writes are buffered by the Dom0.
>>
>> I'm going to give a shot doing the initialization from the DomU (just
>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
>>
>> This is all with bonnie++ -d /mnt -f -u root:root
>>
>> Anyone seen this, or have any ideas?
>>
>> Is any additional latency provided by the xen virtual block device
>> causing a degradation in TCP performance (i.e. a window size or
>> delayed ACK problem) or is the buffering also causing pain? I'm going
>> to keep looking, but I thought I'd ask all of you.
>>
>> Cheers
>>
>> cc
>>
>> --
>> Chris Chen <muffaleta@xxxxxxxxx>
>> "I want the kind of six pack you can't drink."
>> -- Micah
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-users
>>
>
>
>
> --
> Barry van Someren
> ---------------------------------------
> Email: barry@xxxxxxxxxxxxxxx
> Email: goltharnl@xxxxxxxxx
> Linked in: http://www.linkedin.com/in/barryvansomeren
>
--
Chris Chen <muffaleta@xxxxxxxxx>
"I want the kind of six pack you can't drink."
-- Micah
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home