WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] DomU PV network problems

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] DomU PV network problems
From: Sauro Saltini <saltini@xxxxxx>
Date: Wed, 08 Sep 2010 17:54:14 +0200
Delivery-date: Wed, 08 Sep 2010 09:10:39 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4C8780B3.3050207@xxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <4C86AC12.90902@xxxxxx> <AANLkTimXJNtdND61eEzd5CFiVeQDK9XeP9MPgYfHht4O@xxxxxxxxxxxxxx> <4C8780B3.3050207@xxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100121 Lightning/1.0b1 Thunderbird/3.0.1

Some news about the problem...

after receiving some indirect Matej suggestions from Mahdavian, I've looked into the checksum problem mentioned here:
http://wiki.xensource.com/xenwiki/XenFaq#head-4ce9767df34fe1c9cf4f85f7e07cb10110eae9b7

This time I've tried to connect to an http service on a phisical lan host (telnet 192.168.99.202 80) from the testing DomU (10.0.0.102)

here are the results of tcpdump -nvv on the various elements of the lan connected bridge in Dom0 during the connection tests (192.168.99.88 is the fw 'external' IP correctly set by SNAT rule) :

ON vif1.1 - DomU side of bridge :

tcpdump -nvvi vif1.1
tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes 17:31:07.423789 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.99.202 tell 192.168.99.88, length 28 17:31:07.423889 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.99.202 is-at 00:09:6b:89:d0:8a, length 46 17:31:07.423911 IP (tos 0x10, ttl 63, id 3345, offset 0, flags [DF], proto TCP (6), length 60) 192.168.99.88.56168 > 192.168.99.202.80: Flags [S], cksum 0x8d22 (correct), seq 244003632, win 5840, options [mss 1460,sackOK,TS val 474657 ecr 0,nop,wscale 5], length 0
....

here the cksum field is correct


ON br1 - the bridge itself:

tcpdump -nvvi br1
tcpdump: listening on br1, link-type EN10MB (Ethernet), capture size 96 bytes 17:38:13.454259 IP (tos 0x10, ttl 63, id 40899, offset 0, flags [DF], proto TCP (6), length 60) 192.168.99.88.54824 > 192.168.99.202.80: Flags [S], cksum 0x5545 (correct), seq 2632204777, win 5840, options [mss 1460,sackOK,TS val 581162 ecr 0,nop,wscale 5], length 0

again the cksum field is correct


ON peth0 - the phisical NIC :
tcpdump -nvvi peth0 port not 22
tcpdump: WARNING: peth0: no IPv4 address assigned
tcpdump: listening on peth0, link-type EN10MB (Ethernet), capture size 96 bytes 17:40:48.397392 IP (tos 0x10, ttl 63, id 59382, offset 0, flags [DF], proto TCP (6), length 60) 192.168.99.88.44172 > 192.168.99.202.80: Flags [S], cksum 0x48a2 (incorrect -> 0x05a8), seq 774410895, win 5840, options [mss 1460,sackOK,TS val 619897 ecr 0,nop,wscale 5], length 0

Here the cksum field is INCORRECT thus the packet is probably refused !

I've tried to switch off all forms of offloading from peth0 and also from eth0 and eth1 in fw DomU (ethtool -K peth0 tx off rx off sg off tso off ufo off gso off) but the result is the same !

I'm also still confused about the reason why icmp packets (pings) don't even show up on vif1.1 while tcp connection tries do... any ideas ?

What next ? Do I need to try a completely new xenified kernel ?

Many thanks in advance to all...

Sauro.



On 08/09/2010 14:25, Sauro Saltini wrote:

On 08/09/2010 10:42, Fajar A. Nugraha wrote:
On Wed, Sep 8, 2010 at 4:18 AM, Sauro Saltini<saltini@xxxxxx>  wrote:
My current configuration is :
- Xen 4.0.0
- DOM0 kernel 2.6.31.13
- DOMU kernels either 2.6.31.13 or 2.6.36-rc3 (vanilla)
As soon as I've converted the fw DomU's to a PV domain (using either
2.6.31.13 or 2.6.36-rc3 kernels with PV drivers) something changed in a
weird way...
Do you mean convert it to PV domain, or keep it as HVM but use PV drivers?
I've converted it to a PV domain (builder=linux)
on Dom0 "tcpdump -nvvi br0" (br0 = external bridge) never shows up any
traffic !
What does tcpdump on domU's vif (by default it's named vifx.y, where x
and are numbers) within dom0 shows?
Does both dom0 and your PV domU has ip forwarding
(/proc/sys/net/ipv4/ip_forward) enabled?
Do you still use ioemu line? What does your domU config look like?

tcpdump on domU's vif shows no traffic, except some arp requests coming from the firewall domU asking for the external pinged address.
Both dom0 and domU has ip_forward activated
I have type=paravirtual for both firewall domU's nics in domU's config

my config is quite simple:

kernel='/boot/vmlinuz-r410-2.6.31.13'
builder='linux'
memory=512
name='fw'
vcpus=1
vif=[ 'type=paravirtual, bridge=br0, mac=00:16:3e:00:00:02',
      'type=paravirtual, bridge=br1, mac=00:16:3e:00:00:20' ]
disk=[ 'drbd:fw,hda,w' ]
on_poweroff='destroy'
on_reboot='restart'
on_crash='restart'
boot='c'
root='/dev/xvda3 ro'
sdl=0
vnc=0

The chosen kernel is the same xenified kernel built from sources and used for dom0


Please help...
Many thanks in advance.

It seems that the packet flow interrupts in some way inside xen's PV netfront/netback drivers (i.e. between vethx.y and vifx.y) but ONLY for packets coming from another domU and forwarded.

As I said i can regularly ping both domU's network and external lan from the fw domU, only the packets coming from the other domU's disappear !

Just for clarity, having :
testing (pinger) domU : 10.0.0.102 /24 (gw 10.0.0.101)
fw domU "internal" nic -eth0- (domU's gateway): 10.0.0.101 /24
fw domU "external" nic -eth1- (bridged to LAN): 192.168.99.88 /24
pinged lan host : 192.168.99.202 /24

On fw domU:
echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

Pinging 192.168.99.202 from FW works, and tcpdump -nvvi on dom0 (both listening on vifx.y or bridge) gives:
...
14:09:40.553104 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.99.88 > 192.168.99.202: ICMP echo request, id 24837, seq 1, length 64 14:09:40.553223 IP (tos 0x0, ttl 64, id 21099, offset 0, flags [none], proto ICMP (1), length 84) 192.168.99.202 > 192.168.99.88: ICMP echo reply, id 24837, seq 1, length 64
...

Pinging 192.168.99.202 from testing domU (10.0.0.102) doesn't work, and tcpdump -nvvi on dom0 (both listening on vifx.y or bridge) gives (every 5 to 10 seconds):
....
14:17:29.922210 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.99.202 tell 192.168.99.88, length 28 14:17:29.922304 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.99.202 is-at 00:09:6b:89:d0:8a, length 46
....

but no icmp traffic at all !


You can try using xenified kernel on domU and see if it helps: patch
from http://code.google.com/p/gentoo-xen-kernel/downloads/list +
kernel 2.6.34.6 (or whatever 2.6.34.x that the patch can cleanly apply
to).



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users