[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-devel] large packet support in netfront driver and guest network throughput
On 2013-9-16 22:21, Wei Liu wrote:
On Fri, Sep 13, 2013 at 05:09:48PM +0000, Anirban Chakraborty wrote:
On Sep 13, 2013, at 4:44 AM, Wei Liu <wei.liu2@xxxxxxxxxx> wrote:
On Thu, Sep 12, 2013 at 05:53:02PM +0000, Anirban Chakraborty wrote:
Hi All,
I am sure this has been answered somewhere in the list in the past, but I can't
find it. I was wondering if the linux guest netfront driver has GRO support in
it. tcpdump shows packets coming in with 1500 bytes, although the eth0 in dom0
and the vif corresponding to the linux guest in dom0 is showing that they
receive large packet:
In dom0:
eth0 Link encap:Ethernet HWaddr 90:E2:BA:3A:B1:A4
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
tcpdump -i eth0 -nnvv -s 1500 src 10.84.20.214
17:38:25.155373 IP (tos 0x0, ttl 64, id 54607, offset 0, flags [DF], proto TCP
(6), length 29012)
10.84.20.214.51041 > 10.84.20.213.5001: Flags [.], seq 276592:305552, ack
1, win 229, options [nop,nop,TS val 65594025 ecr 65569225], length 28960
vif4.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
tcpdump -i vif4.0 -nnvv -s 1500 src 10.84.20.214
17:38:25.156364 IP (tos 0x0, ttl 64, id 54607, offset 0, flags [DF], proto TCP
(6), length 29012)
10.84.20.214.51041 > 10.84.20.213.5001: Flags [.], seq 276592:305552, ack
1, win 229, options [nop,nop,TS val 65594025 ecr 65569225], length 28960
In the guest:
eth0 Link encap:Ethernet HWaddr CA:FD:DE:AB:E1:E4
inet addr:10.84.20.213 Bcast:10.84.20.255 Mask:255.255.255.0
inet6 addr: fe80::c8fd:deff:feab:e1e4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
tcpdump -i eth0 -nnvv -s 1500 src 10.84.20.214
10:38:25.071418 IP (tos 0x0, ttl 64, id 15074, offset 0, flags [DF], proto TCP
(6), length 1500)
10.84.20.214.51040 > 10.84.20.213.5001: Flags [.], seq 17400:18848, ack 1,
win 229, options [nop,nop,TS val 65594013 ecr 65569213], length 1448
Is the packet on transfer from netback to net front is segmented into MTU size?
Is GRO not supported in the guest?
Here is what I see in the guest, iperf server running in guest and iperf
client running in Dom0. Tcpdump runs with the rune you provided.
10.80.238.213.38895 > 10.80.239.197.5001: Flags [.], seq
5806480:5818064, ack 1, win 229, options [nop,nop,TS val 21968973 ecr
21832969], length 11584
This is a upstream kernel. The throughput from Dom0 to DomU is ~7.2Gb/s.
Thanks for your reply. The tcpdump was captured on dom0 of the guest [at both
vif and the physical interfaces] , i.e. on the receive path of the server.
iperf server was running on the guest (10.84.20.213) and the client was at
another guest (on a different server) with IP 10.84.20.214. The traffic was
between two guests, not between dom0 and the guest.
I am seeing extremely low throughput on a 10Gb/s link. Two linux guests (Centos
6.4 64bit, 4 VCPU and 4GB of memory) are running on two different XenServer
6.1s and iperf session between them shows at most 3.2 Gbps.
XenServer might use different Dom0 kernel with their own tuning. You can
also try to contact XenServer support for better idea?
XenServer 6.1 is running 2.6.32.43 kernel. Since the issue is in netfront
driver, as it appears from the tcpdump, thats why I thought I post it here.
Note that checksum offloads of the interfaces (virtual and physical) were not
even touched, the default setting (which was set to on) was used.
In general, off-host communication can be affected by various things. It
would be quite useful to identify the bottleneck first.
Try to run:
1. Dom0 to Dom0 iperf (or you workload)
2. Dom0 to DomU iperf
3. DomU to Dom0 iperf
I tried dom0 to dom0 and I got 9.4 Gbps, which is what I expected (with GRO
turned on in the physical interface). However, when I run guest to guest,
things fall off. Is large packet not supported in netfront? I thought
otherwise. I looked at the code and I do not see any call to
napi_gro_receive(), rather it is using netif_receive_skb(). netback seems to be
sending GSO packets to the netfront, but it is being segmented to 1500 byte (as
it appears from the tcpdump).
OK, I get your problem.
Indeed netfront doesn't make use of GRO API at the moment.
This is true.
But I am wondering why large packet is not segmented into mtu size with
upstream kernel? I did see large packets with upsteam kernel on receive
guest(test between 2 domus on same host).
Thanks
Annie
I've added
this to my list to work on. I will keep you posted when I get to that.
Thanks!
Wei.
In order to get line rate, you need to at least get line rate from Dom0
to Dom0 IMHO. 10G/s line rate from guest to guest has not yet been
achieved at the momentâ
What is the current number, without VCPU pinning etc. for 1500 byte MTU? I am
getting 2.2-3.2 Gbps for 4VCPU guest with 4GB of memory. It is the only vm
running on that server without any other traffic.
-Anirban
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
