[Xen-users] Xen's xenbr0 interface eating packets between domUs?

["Jan Mulders" <lastchancehotel@xxxxxxxxx>]

Hello all.

Having a little issue with a Xen setup I'm running. my domU's don't seem to be able to contact one another...


Can anyone shed some light on where my packets are going? They vanish between the domU sending the response and the xenbr0 on the dom0.

Here's some details - IP addresses obfuscated with 11.22.33 for safety.

forest = dom0
squirrel = gameserver domU
tree = openvpn domU
rock = webserver domU (not mentioned here except for 'it has the same problem')
here's some dom0 ip configuration:

[root@forest ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:02:B3:E9:A7:EB
          inet addr:11.22.33.161  Bcast: 
11.22.33.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:172153 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28522 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15218006 (14.5 MiB)  TX bytes:11912815 (11.3 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:
255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:512496 (500.4 KiB)  TX bytes:512496 (500.4 KiB)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:17906458 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17406759 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3001638846 (
2.7 GiB)  TX bytes:629520683 (600.3 MiB)

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:28522 errors:0 dropped:0 overruns:0 frame:0
          TX packets:172154 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:11912815 (11.3 MiB)  TX bytes:15218084 (14.5 MiB)

vif9.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12558483 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13041996 errors:0 dropped:188 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:65048010 (62.0 MiB)  TX bytes:138801861 (132.3 MiB)

vif15.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1357 errors:0 dropped:0 overruns:0 frame:0
          TX packets:110982 errors:0 dropped:47 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:752723 (735.0 KiB)  TX bytes:10116656 (9.6 MiB)

vif17.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3762543 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3559047 errors:0 dropped:2167 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:477432410 (455.3 MiB)  TX bytes:1187204888 (1.1 GiB)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:392477 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:144771106 (138.0 MiB)  TX bytes:0 (0.0 b)

I'm trying to send udp packets to the gameserver (squirrel) from a machine behind the VPN machine (tree). The forward route (me to gameserver) seems to be fine - but the return trip? Not so good.

11.22.33.163 is me, 11.22.33.138 is the gameserver (squirrel).
Here's what I get from tcpdump's on each machine:

First of all, here's the gameserver.

[root@squirrel ~]# tcpdump -i eth0 src 11.22.33.163 or dst 11.22.33.163 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
16:23:24.275874 IP 11.22.33.163.3724 > 11.22.33.138.27015: UDP, length 25
16:23:24.276129 IP 11.22.33.138.27015 > 11.22.33.163.3724 : UDP, length 109
16:23:31.825710 IP 11.22.33.163.3732 > 11.22.33.138.27015: UDP, length 25
16:23:31.835973 IP 11.22.33.138.27015 > 11.22.33.163.3732: UDP, length 109
16:23:31.839778 IP 11.22.33.163.3733 > 11.22.33.138.27015
: UDP, length 25
16:23:31.850976 IP 11.22.33.138.27015 > 11.22.33.163.3733: UDP, length 109
16:23:36.835129 arp who-has 11.22.33.163 tell 11.22.33.138
16:23:36.835800 arp reply 11.22.33.163 is-at 00:16:3e:1d:28:ee

8 packets captured
8 packets received by filter
0 packets dropped by kernel
[root@squirrel ~]#

As you can see, the server responded correctly to all three requests.

Here is the Xen dom0 (forest), sniffing at the bridge:

[root@forest ~]# tcpdump -i xenbr0 src 11.22.33.163
 or dst 195.178.107.163 -n
tcpdump: WARNING: xenbr0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xenbr0, link-type EN10MB (Ethernet), capture size 96 bytes
21:23:24.449795 IP 11.22.33.163.3724 > 11.22.33.138.27015: UDP, length 25
21:23:32.001248 IP 11.22.33.163.3732 > 11.22.33.138.27015: UDP, length 25
21:23:32.015363 IP 11.22.33.163.3733 > 11.22.33.138.27015
: UDP, length 25
21:23:37.012320 arp reply 11.22.33.163 is-at 00:16:3e:1d:28:ee

28 packets captured
35 packets received by filter
0 packets dropped by kernel
[root@forest
 ~]#

Now this is interesting. The replies aren't making it onto the xenbr0 interface! the who-has arp request got gobbled by the filter but it'll be there.

Here is the VPN domU (tree), sniffing at its eth0.

[root@tree ~]# tcpdump -i eth0 src 11.22.33.138 or src 195.178.107.163 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
21:23:24.403375 IP 11.22.33.138.27015 > 11.22.33.163.3724: UDP, length 109
21:23:31.964292 IP 11.22.33.138.27015 > 11.22.33.163.3732 : UDP, length 109
21:23:31.979393 IP 11.22.33.138.27015 > 11.22.33.163.3733: UDP, length 109
21:23:36.943329 arp reply 11.22.33.138 is-at 00:16:3e:39:84:0f
21:23:36.964097 arp who-has 
11.22.33.163 tell 11.22.33.138

5 packets captured
27 packets received by filter
0 packets dropped by kernel
[root@tree ~]#

And predictably, the replies don't make it to the VPN domU (tree).

The only rules in the dom0 which could possibly stop packets systematically are:
[root@forest ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            PHYSDEV match --physdev-in vif9.0
ACCEPT     all  --  anywhere             anywhere            PHYSDEV match --physdev-in 
vif15.0
ACCEPT     all  --  anywhere             anywhere            PHYSDEV match --physdev-in vif17.0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[root@forest ~]#


People out on the internet can connect to my server no problem, and I can access the internet fine, too... but accessing this machine within the Xen bridge doesn't seem to work.

Some Xen configs:

Tree:

kernel = "/boot/vmlinuz-2.6.16-xen0-100hz"
memory = 64
name = "tree"
vif = [ '' ]
ip = "11.22.33.139"
gateway = "
11.22.33.1"
netmask = "255.255.255.0"
disk = ['phy:/dev/vg01/vol01,sda1,w','phy:/dev/vg01/vol02,sda2,w']
root = "/dev/sda1 ro"

kernel = "/boot/vmlinuz-2.6.16-xen0-1000hz"
memory = 128
name = "squirrel"
vif = [ '' ]
ip = "11.22.33.138"
gateway = "
11.22.33.1"
netmask = "255.255.255.0"
disk = ['phy:/dev/vg01/vol03,sda1,w','phy:/dev/vg01/vol04,sda2,w']
root = "/dev/sda1 ro"

I also have another domU called 'rock', mentioned earlier, that doesn't respond directly from inside the bridge; yet responds fine when the request comes from outside the Xen bridge (ie, from the internet).

Can anyone shed some light on where my packets are going?








_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users