[Xen-bugs] [Bug 1132] New: Xen Dom0 TCP low throughput with TC-TBF traffic shaping

[bugzilla-daemon@xxxxxxxxxxxxxxxxxxx]

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1132

           Summary: Xen Dom0 TCP low throughput with TC-TBF traffic shaping
           Product: Xen
           Version: 3.0.3
          Platform: x86-64
        OS/Version: Linux-2.6
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Unspecified
        AssignedTo: xen-bugs@xxxxxxxxxxxxxxxxxxx
        ReportedBy: jsong@xxxxxxxxxxxxxxxx
                CC: jsong@xxxxxxxxxxxxxxxx


I got very low TCP throughput with iperf and netperf while using TC-TBF (Linux
native TC token bucket traffic shaping) on the physical Giga NIC.

I attached Linux TC TBF (Linux native Traffic Control token bucket filter) to
the physical NICs on two REHL5 Xen boxes (eth1 on machine ios3 and devmsunf):

tc qdisc add dev eth1 root tbf rate 1000mbit burst 100mb latency 100ms

Three Linux boxes connect to a HP Giga switch. Netperf TCP traffic was sent
from the two Xen boxes ( devmsunf and ios3) to a third network client running
REHL5 (ios2) from PNIC to PNIC in two separate tests. ios3 gave a throughput of
940Mbps, while devmsunf gave a throughput of only a few hundred Kbps. Netperf
UDP tests gave 940Mbps from both boxes. Without TC-TBF, both boxes gave 940Mbps
TCP throughput. It seemed that Xen, TCP and TC-TBF is a bad combination. I
don't know why two Xen boxes giving different TCP throughput. I tried to
increase the TCP window (TCP Send Buffer Size) from OS default 16K to 64K, but
no improvement in TCP throughput on devmsunf.

I also have the same problem when I used Xen Dom0 running netserver (netperf
receiver). With default TCP window size, the netperf TCP traffic from a REHL5
CN to the Xen receiver was unstable, varying from 800Mbps to a total throttling
state (a few kbps). After setting the netserver TCP window size to 64K (default
16K) on the Xen box, the TCP traffic performed good steady at 870Mbps.
Increasing TCP window via setsockopt() in the netperf/iperf application code
before opening TCP connection seemed to fix this problem on Xen kernel.
However, setting global TCP TCP receiving and sending buffer via sysctl in
/etc/sysctl.conf like the following didn't work.

# increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

The iperf/netperf application seemed to pick up the adjusted TCP window, but no
improvement in throughput was observed. I am not sure whether Linux auto-tuning
on the send buffer is overwriting my settings or not. The problem seems to be
also related to the TCP round trip time (The RTT is around 0.1ms in my
network).

FYI, I am using Xen version 3.0.3-rc5-8.el5 on Red Hat Enterprise Linux Server
(2.6.18-8.el5xen).

Has anybody experienced similar problem? Any help is very welcomed.
Thanks in advance!

Jialin


-- 
Configure bugmail: 
http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-bugs