[Xen-users] Xen network-bridge problem

Hi,

Still facing this problem, can anyone take a look at the latest info I have?

The full story is below.

eth0 interface (Xen Virtual Interface) on the VM is showing growing Rx bytes on every ifconfig command, but Tx bytes is not moving.

Also: ‘qemu-dm.log’ maybe has something:

domid: 1

Warning: vlan 0 is not connected to host network

Watching /local/domain/0/device-model/1/logdirty/next-active

Watching /local/domain/0/device-model/1/command

char device redirected to /dev/pts/1

/builddir/build/BUILD/xen-3.4.0/tools/ioemu-dir/hw/xen_blktap.c:628: Init blktap pipes

xen be: console-0: backend state: Initialising -> InitWait

xen be: console-0: frontend not ready, ignoring

xen be: console-0: bind evtchn port 21

xen be: console-0: ring mfn 1277431, remote port 2, local port 21, limit 1048576

xen be: console-0: backend state: InitWait -> Connected

xs_read(): vncpasswd get error. /vm/edc7ed59-91d2-1c7f-9523-dd2f102eb74e/vncpasswd.

xen be: console-0: backend update: state

xen be: console-0: backend update: hotplug-status

xen be: console-0: backend update: state

xen be: console-0: frontend update: vnc-port

xen be: console-0: frontend update: tty

xen be: console-0: backlog piling up, nobody listening?

If I try to ping the vm 10.2.0.55 inside xen(102.0.52) and monitor it with tcpdump –nvvi xenbr0 10.2.0.110 | grep –i 10.2.0.55

I see:

# tcpdump: listening on xenbr0, link-type EN10MB (Ethernet), capture size 96 bytes

19:49:25.256958 arp who-has 10.2.0.55 tell 10.2.0.110

19:49:25.257378 arp reply 10.2.0.55 is-at 00:16:3e:07:03:49

So it is discoverable isn’t it? Why does the Tx byte stands still?

Also following this immediately afterwards with ‘ip neigh’ on the vm(10.2.0.55) returns

10.2.0.110 dev eth0 lladr 00:26:..etc STALE which I understand is still usable, right?

Thanks!

=================================

My Xen includes 2 guests. Xen itself (10.2.0.52) gets free access to the outside world and to its guests.

Both guests however (10.2.0.54/10.2.0.55) see each other but stay under house arrest!

Not a single ping manages to go past the bridge (xenbr0) and get an answer from the default gateway (10.2.0.254.)

On the guest I keep seeing 'Destination Host Unreachable', i.e.:

#traceroute 10.2.0.254 comes back with the !H message.

I have pasted all the info I gathered in attempt to understand who locks the guests in, but will be happy to add any further output required to get this resolved.

I followed so many manuals and links, one that seriously caught my attention had to do with MAC addresses issue. Could this be my case?.

Many thanks for any assistance,

Liron.

#tcpdump -nvvi xenbr0 | grep -i 10.2.0.55

---------------------------------------------

11:19:41.548806 arp who-has 10.2.0.254 tell 10.2.0.55

11:19:41.549059 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.2.0.55 > 10.2.0.254: ICMP echo request, id 19463, seq 19, length 64

#tcpdump -nvvi vif1.0 host 10.2.0.55

------------------------------------

11:21:23.403063 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.2.0.55 > 10.2.0.254: ICMP echo request, id 22279, seq 7, length 64

#tcpdump -vv -n -c 1000 | grep -i 10.2.0.254

--------------------------------------------

11:22:46.256389 arp who-has 10.2.0.254 tell 10.2.0.55

11:22:46.256662 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.2.0.55 > 10.2.0.254: ICMP echo request, id 23047, seq 8, length 64

# iptables -L -vv -n

--------------------

Chain INPUT (policy ACCEPT 10869 packets, 769K bytes)

pkts bytes target prot opt in out source destination

0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8003

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED PHYSDEV match --physdev-out vif1.0

0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif1.0 udp spt:68 dpt:67

0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED PHYSDEV match --physdev-out vif1.0

0 0 ACCEPT all -- * * 10.2.0.55 0.0.0.0/0 PHYSDEV match --physdev-in vif1.0

Chain OUTPUT (policy ACCEPT 8294 packets, 2149K bytes)

pkts bytes target prot opt in out source destination

----------------------------------------------

As in http://wiki.xensource.com/xenwiki/XenNetworking#head-602e26cd4a03b992f3938fe1bea03fa0fea0ed8b

I tried:

#echo "1" > /proc/sys/net/ipv4/ip_forward

I also tried:

iptables -A FORWARD -m physdev --physdev-in eth0 --physdev-out '!' eth0 -j ACCEPT

iptables -A FORWARD -m physdev --physdev-out eth0 --physdev-in '!' eth0 -j ACCEPT

Still Nothing!

----------------------------------------------

# cat /etc/xen/xend-config.sxp | grep -v "^#" | grep "[a-z]"

-----------------------------------------------------------

(logfile /var/log/xen/xend.log)

(loglevel DEBUG)

(network-script network-bridge)

(vif-script vif-bridge)

(dom0-min-mem 196)

(enable-dom0-ballooning yes)

(dom0-cpus 0)

(vnc-listen '0.0.0.0')

(vncpasswd '')

(xend-domains-lock-path /opt/ovs-agent-2.3/utils/dlm.py)

-----------------------------------------------

# brctl show

bridge name bridge id STP enabled interfaces

xenbr0 8000.0050568b3ae4 no vif1.0

eth0

--------------------------------------------------------------------

# xm list

Name ID Mem VCPUs State Time(s)

Domain-0 0 564 2 r----- 67.2

SiebelApp 1 1024 1 -b---- 32.6

---------------------------------------------------------------------

#xm network-list 1

-------------------

Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path

0 0 00:16:3e:07:03:49 0 4 10 815 /816 /local/domain/0/backend/vif/1/0

--------------------

# xm network-list 1 -l

((mac 00:16:3e:07:03:49)

(handle 0)

(protocol x86_32-abi)

(backend-id 0)

(state 4)

(backend /local/domain/0/backend/vif/1/0)

(tx-ring-ref 815)

(rx-ring-ref 816)

(event-channel 10)

(request-rx-copy 1)

(feature-rx-notify 1)

(feature-sg 1)

(feature-gso-tcpv4 1)

)

-------------------------------

# ifconfig -a --> ON 10.2.0.52 (Domain-0)

------------------------------------

eth0 Link encap:Ethernet HWaddr 00:50:56:8B:3A:E4

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:18937 errors:0 dropped:0 overruns:0 frame:0

TX packets:13043 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1484679 (1.4 MiB) TX bytes:2669625 (2.5 MiB)

Interrupt:17 Base address:0x2000

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:124 errors:0 dropped:0 overruns:0 frame:0

TX packets:124 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:17724 (17.3 KiB) TX bytes:17724 (17.3 KiB)

vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:136 errors:0 dropped:0 overruns:0 frame:0

TX packets:5272 errors:0 dropped:154 overruns:0 carrier:0

collisions:0 txqueuelen:32

RX bytes:12878 (12.5 KiB) TX bytes:457533 (446.8 KiB)

xenbr0 Link encap:Ethernet HWaddr 00:50:56:8B:3A:E4

inet addr:10.2.0.52 Bcast:0.0.0.0 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:18955 errors:0 dropped:0 overruns:0 frame:0

TX packets:12891 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1224607 (1.1 MiB) TX bytes:2652734 (2.5 MiB)

--------------------------------------

# ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:50:56:8b:3a:e4 brd ff:ff:ff:ff:ff:ff

3: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue

link/ether 00:50:56:8b:3a:e4 brd ff:ff:ff:ff:ff:ff

4: vif1.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 32

link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

---------------------------------------------------------------

#vi /etc/sysconfig/network-scripts/ifcfg-eth0 (ON Domain-0 --> 10.2.0.52)

--------------------------------------------

# Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]

DEVICE=eth0

BOOTPROTO=none

BROADCAST=10.2.0.255

HWADDR=00:50:56:8b:3a:e4

IPADDR=10.2.0.52

NETMASK=255.255.255.0

NETWORK=10.2.0.0

GATEWAY=10.2.0.254

TYPE=Ethernet

---------------------------------------------

# arping -b -c 1 -I xenbr0 10.2.0.55 (to the Guest VM)

-------------------------------------

ARPING 10.2.0.55 from 10.2.0.52 xenbr0

Unicast reply from 10.2.0.55 [00:16:3E:07:03:49] 1.684ms

Sent 1 probes (1 broadcast(s))

Received 1 response(s)

----------------------------------------------------------

[root@LinuxDB network-scripts]# cd /etc/xen/scripts/

[root@LinuxDB scripts]# ./network-bridge status

============================================================

4: vif1.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 32

link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

3: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue

link/ether 00:50:56:8b:3a:e4 brd ff:ff:ff:ff:ff:ff

inet 10.2.0.52/24 scope global xenbr0

bridge name bridge id STP enabled interfaces

xenbr0 8000.0050568b3ae4 no vif1.0

eth0

10.2.0.0/24 dev xenbr0 proto kernel scope link src 10.2.0.52

default via 10.2.0.254 dev xenbr0

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

10.2.0.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr0

0.0.0.0 10.2.0.254 0.0.0.0 UG 0 0 0 xenbr0

============================================================

In my vm.cfg I added:

vif = ['ip=10.2.0.55,mac=00:16:3e:07:03:49,bridge=xenbr0']

===============================================================

#brctl showmacs xenbr0 (It's actually a very long list out of which I cut the last section that has local = 'yes')

----------------------

1 00:50:56:8b:3a:e4 yes 0.00

1 00:50:56:8b:40:87 no 20.43

1 00:50:56:8b:48:08 no 260.19

1 00:50:56:8b:48:11 no 152.62

1 00:50:56:8b:50:d4 no 215.24

1 00:50:56:8b:68:9a no 27.69

1 00:50:56:b5:07:e2 no 170.61

1 00:e0:07:03:40:88 no 5.22

1 00:e0:4d:8b:8a:89 no 80.37

1 0a:01:00:00:00:00 no 1.23

2 fe:ff:ff:ff:ff:ff yes 0.00

I'm also adding a part of the "/var/log/xen/xend.log" In case it helps. If you need to see more from it, i'd be happy to add, but I don't see any errors there.

--------------------------------------------------------

[2010-04-30 11:13:23 2993] INFO (XendDomainInfo:2180) createDevice: vif : {'ip': '10.2.0.55', 'mac': '00:16:3e:07:03:49', 'uuid': 'e475dc37-4529-423f-157a-2b813c14afbc', 'bridge': 'xenbr0'}

[2010-04-30 11:13:23 2993] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3e:07:03:49', 'handle': '0', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.

[2010-04-30 11:13:23 2993] DEBUG (DevController:97) DevController: writing {'bridge': 'xenbr0', 'domain': 'SiebelApp', 'handle': '0', 'uuid': 'e475dc37-4529-423f-157a-2b813c14afbc', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '10.2.0.55', 'mac': '00:16:3e:07:03:49', 'frontend-id': '1', 'state': '1', 'online': '1', 'frontend': '/local/domain/1/device/vif/0'} to /local/domain/0/backend/vif/1/0.

[2010-04-30 11:13:23 2993] INFO (image:391) spawning device models: /usr/lib/xen/bin/qemu-dm ['/usr/lib/xen/bin/qemu-dm', '-d', '1', '-domain-name', 'SiebelApp', '-videoram', '4', '-vnc', '0.0.0.0:0', '-vncunused', '-serial', 'pty', '-M', 'xenpv']

[2010-04-30 11:13:23 2993] INFO (image:440) device model pid: 3367

[2010-04-30 11:13:23 2993] DEBUG (XendDomainInfo:3080) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': '07382902-6cc8-ee56-18ad-f6d759aa3788', 'on_reboot': 'restart', 'start_time': '1272615203.93', 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': "(linux (kernel ) (videoram 4) (device_model /usr/lib/xen/bin/qemu-dm) (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 3221225472) (GUEST_OS linux) (HYPERCALL_PAGE 3225423872) (LOADER generic) (PAE_MODE yes) (ENTRY 3225419776) (XEN_VERSION xen-3.0)))", 'name': 'SiebelApp'}

[2010-04-30 11:13:23 2993] INFO (image:528) waiting for sentinel_fifo

[2010-04-30 11:13:23 2993] DEBUG (XendDomainInfo:1640) Storing domain details: {'console/ring-ref': '1277431', 'image/entry': '3225419776', 'console/port': '2', 'store/ring-ref': '1277432', 'image/loader': 'generic', 'vm': '/vm/07382902-6cc8-ee56-18ad-f6d759aa3788', 'control/platform-feature-multiprocessor-suspend': '1', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '1048576', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'console/limit': '1048576', 'image/paddr-offset': '3221225472', 'image/hypercall-page': '3225423872', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'ioemu', 'image/features/auto-translated-physmap': '1', 'name': 'SiebelApp', 'domid': '1', 'image/xen-version': 'xen-3.0', 'store/port': '1'}

NOTICE: This e-mail message and any attachments may contain confidential, proprietary, or privileged information. If you are not the intended recipient, please notify us immediately by return e-mail, delete this message, and destroy all physical and electronic copies. Thank you.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

WARNING - OLD ARCHIVES

xen-users

[Xen-users] Xen network-bridge problem