WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] strangeness with high BDP tests

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] strangeness with high BDP tests
From: Diwaker Gupta <diwakergupta@xxxxxxxxx>
Date: Fri, 1 Apr 2005 20:14:47 -0800
Delivery-date: Sat, 02 Apr 2005 04:14:50 +0000
Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=q1RqqD9FSRYQjXNi9YurfwkJWiypKm7VcL+GtWqUSU/zLkNiaMyLb+mpuZx9RrA5Xp89F7ScQ4RQ9AncHvJa12HYQ8D/HRkWg85KTd/F3sfVqRJbf24Nwp+uE17UuAOBp2wilJreu0hHdSopu7nRutnCYhDfrpZ9AKpXp673VOY=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <1b0b4557050401171624fdc24f@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <1b0b4557050401171624fdc24f@xxxxxxxxxxxxxx>
Reply-to: Diwaker Gupta <diwakergupta@xxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Hi everyone,

First, let me establish the baseline here:
All default settings, no modifications to any sysctls. Same amount of
RAM to dom0 and VM. (note that by default, TCP BiC is on). Test across
a low latency cluster, everything on the same gigabit switch. I'm
using Xen 2.0.3. I'm using netperf for all my test.

Between dom0 to dom0 on two machines in the cluster, I can consistenly
get ~930Mbps. Between VM to VM on the same two machines, I can get
between 730 to  850 Mbps, but there's a lot more variation.

So far so good.

Now, I modify the TCP buffer sizes (both on dom0 and VM) thus:
net.ipv4.tcp_rmem = 4096        87380   8388608
net.ipv4.tcp_wmem = 4096        65536   8388608
net.ipv4.tcp_mem = 24576        32768   49152
net.core.rmem_default = 112640
net.core.wmem_default = 112640
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_bic_low_window = 14
net.ipv4.tcp_bic_fast_convergence = 1
net.ipv4.tcp_bic = 1

Now, between dom0 to dom0 on 2 machines, I can get consistenly get
880Mbps. And between VM to VM, I can get around 850Mbps. So far so
good.

But now comes the really interesting part. So far, these machines were
talking over the switch directly. Now I direct all traffic through a
dummynet router (on the same switch).  The pipe connecting the two is
set to 500Mbps with an RTT of 80ms.

Here are the results for dom0 to dom0 tests:

== Single flow, 10 seconds ==
[dgupta@sysnet03]$ netperf -H sysnet08
TCP STREAM TEST to sysnet08
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  65536  65536    10.11     158.55

== Single flow, 80 seconds ==
[dgupta@sysnet03]$ netperf -H sysnet08 -l 80
TCP STREAM TEST to sysnet08
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  65536  65536    80.72     344.20

== 50 flows, 80 seconds ==

 87380  65536  65536    80.14       4.93
 87380  65536  65536    80.18       9.37
 87380  65536  65536    80.21      10.13
 87380  65536  65536    80.22       9.11
 87380  65536  65536    80.19       9.45
 87380  65536  65536    80.22       5.06
 87380  65536  65536    80.15       9.38
 87380  65536  65536    80.20       9.98
 87380  65536  65536    80.23       3.70
 87380  65536  65536    80.20       9.14
 87380  65536  65536    80.18       8.85
 87380  65536  65536    80.16       8.96
 87380  65536  65536    80.21       9.91
 87380  65536  65536    80.18       9.46
 87380  65536  65536    80.17       9.38
 87380  65536  65536    80.18       9.82
 87380  65536  65536    80.15       7.22
 87380  65536  65536    80.16       8.64
 87380  65536  65536    80.26      10.60
 87380  65536  65536    80.22       9.33
 87380  65536  65536    80.24       8.88
 87380  65536  65536    80.22       9.54
 87380  65536  65536    80.19       9.65
 87380  65536  65536    80.20       9.70
 87380  65536  65536    80.24       9.43
 87380  65536  65536    80.19       8.10
 87380  65536  65536    80.21       9.31
 87380  65536  65536    80.18       9.08
 87380  65536  65536    80.19       9.24
 87380  65536  65536    80.27       9.91
 87380  65536  65536    80.28       9.67
 87380  65536  65536    80.24       9.50
 87380  65536  65536    80.28       9.70
 87380  65536  65536    80.24      10.09
 87380  65536  65536    80.31       4.55
 87380  65536  65536    80.28       5.93
 87380  65536  65536    80.25       9.55
 87380  65536  65536    80.32       5.60
 87380  65536  65536    80.35       6.29
 87380  65536  65536    80.27       4.75
 87380  65536  65536    80.40       6.51
 87380  65536  65536    80.39       6.38
 87380  65536  65536    80.40      10.12
 87380  65536  65536    80.53       4.62
 87380  65536  65536    80.67      16.53
 87380  65536  65536    81.10       4.53
 87380  65536  65536    82.21       1.93
 87380  65536  65536    80.09       9.43
 87380  65536  65536    80.10       9.14
 87380  65536  65536    80.13       9.88
[~]
[dgupta@sysnet03]$ awk '{sum+=$5} END {print sum,NR,sum/NR}' dom0-dom0-50.dat
419.96 50 8.3992

This the aggregate and average per flow. Now I run the same test from VM to VM:

== Single flow, 10 seconds ==
root@tg3:~# netperf -H 172.19.222.101
TCP STREAM TEST to 172.19.222.101
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  65536  65536    10.15      22.30

== Single flow, 80 seconds ==
root@tg3:~# netperf -H 172.19.222.101 -l 80
TCP STREAM TEST to 172.19.222.101
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  65536  65536    80.17      76.96

== 50 flows, 80 seconds ==
 tee vm-vm-50.dati=0;i<50;i++));do netperf -P 0 -H 172.19.222.101 -l 80 & done |
 87380  65536  65536    80.09       8.50
 87380  65536  65536    80.08       6.46
 87380  65536  65536    80.19       7.33
 87380  65536  65536    80.20       7.29
 87380  65536  65536    80.20       5.86
 87380  65536  65536    80.23       8.40
 87380  65536  65536    80.22       8.55
 87380  65536  65536    80.22       7.34
 87380  65536  65536    80.29       6.28
 87380  65536  65536    80.28       7.23
 87380  65536  65536    80.23       8.56
 87380  65536  65536    80.25       6.60
 87380  65536  65536    80.31       6.99
 87380  65536  65536    80.27       8.22
 87380  65536  65536    80.30       7.41
 87380  65536  65536    80.33       8.21
 87380  65536  65536    80.27       7.94
 87380  65536  65536    80.32       6.54
 87380  65536  65536    80.29       8.58
 87380  65536  65536    80.35       7.37
 87380  65536  65536    80.35       7.09
 87380  65536  65536    80.37       7.23
 87380  65536  65536    80.38       8.31
 87380  65536  65536    80.38       8.18
 87380  65536  65536    80.44       9.11
 87380  65536  65536    80.43       4.95
 87380  65536  65536    80.43       6.48
 87380  65536  65536    80.42       8.11
 87380  65536  65536    80.44       6.74
 87380  65536  65536    80.47       8.76
 87380  65536  65536    80.42       7.68
 87380  65536  65536    80.45       6.10
 87380  65536  65536    80.46       7.47
 87380  65536  65536    80.51       7.37
 87380  65536  65536    80.52       6.78
 87380  65536  65536    80.48       7.31
 87380  65536  65536    80.56       7.55
 87380  65536  65536    80.57       6.85
 87380  65536  65536    80.59       7.53
 87380  65536  65536    80.63       7.01
 87380  65536  65536    80.64       6.78
 87380  65536  65536    80.60       5.76
 87380  65536  65536    80.79       6.63
 87380  65536  65536    80.79       6.29
 87380  65536  65536    80.81       7.54
 87380  65536  65536    80.81       7.22
 87380  65536  65536    80.94       6.54
 87380  65536  65536    80.90       8.02
 87380  65536  65536    81.15       4.22

root@tg3:~# awk '{sum+=$5} END {print sum,NR,sum/NR}' vm-vm-50.dat
361.74 50 7.2348

Note the the terrible performance with single flows. With 50 flows,
the aggregate improves, but is still much worse than the dom0 to dom0
results.

Any ideas why I'm getting such bad performance from the VMs on high
BDP links? I'm willing and interested to help in debugging and fixing
this issue, but I need some leads :)

TIA
--
Diwaker Gupta
http://resolute.ucsd.edu/diwaker

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users