WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

RE: [Xen-users] High availability Xen with bonding

To: Bart Coninckx <bart.coninckx@xxxxxxxxxx>, "xen-users@xxxxxxxxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: RE: [Xen-users] High availability Xen with bonding
From: Eric van Blokland <Eric@xxxxxxxxxxxx>
Date: Mon, 18 Oct 2010 08:11:33 +0200
Accept-language: nl-NL
Acceptlanguage: nl-NL
Cc:
Delivery-date: Mon, 18 Oct 2010 01:08:16 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <201010161537.49007.bart.coninckx@xxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <32D516B498BAE9439B6502E86F31007A01BA76E6DDF3@domain> <201010161537.49007.bart.coninckx@xxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: ActtJ0sKvNDLOpYBRkiczXGDU4tpqABdBLEg
Thread-topic: [Xen-users] High availability Xen with bonding
Hey Bart,

For the completeness of my config:

All interfaces participating in the bridge should have mac address 
"fe:ff:ff:ff:ff:ff" and the bridge forwarding delay should be set to 0.

After the boot, some link settings should be modified with ip link set, like I 
mentioned in my previous mail. The following commands should be issued in a 
setup like mine:

ip link set eth0 arp off
ip link set eth1 arp off

ip link set bond0 multicast off arp off
ip link set xenbr0 multicast off arp off ip link set vif0.0 multicast off arp 
off ip link set vif1.0 multicast off arp off


I've also tried the network-bridge-bonding script, which appears to be working 
fine. It creates a situation similar to my manual one, the main difference is 
that instead of veth0 being renamed to bond0, the bridge is created as bond0 
instead, so no xenbr0, which seems ok to me.

Finally I tried the default network-bridge script, without initializing eth0 
and eth1 on boot. In this config, eth0 and eth1 had pbond0 as master, and I 
brought them up manually after xend had started. Seems to work fine as well.

Now for the "however"...

In each setup, I experienced connection loss, or packet loss to be precise, to 
my Dom1. When this occurs, I see the packets arriving at the bridge, but 
they're not forwarded to vif1.0. I haven't tested what happens to packets from 
Dom1 to the bridge. I've not been able to find the cause of this issue yet. 
Everything seems to be running fine for several minutes and then suddenly ping 
request packets aren't forwarded anymore. Increase of network traffic seems to 
be partly triggering it, however it's not the primary cause. 

I'm going to do some further testing this week, but it seems this *only* 
happens when both eth0 and eth1 are in the bond. If I only use one link, 
everything seems to work fine. When both links are in the bond I see packets 
entering and leaving on each link, whether connectivity to Dom1 is fine or not. 
It's also worth to mention, that there are no connectivity problems with Dom0.

Please keep posted on your progress.

Regards,

Eric

-----Oorspronkelijk bericht-----
Van: Bart Coninckx [mailto:bart.coninckx@xxxxxxxxxx] 
Verzonden: zaterdag 16 oktober 2010 15:38
Aan: xen-users@xxxxxxxxxxxxxxxxxxx
CC: Eric van Blokland
Onderwerp: Re: [Xen-users] High availability Xen with bonding

On Friday 15 October 2010 13:44:42 Eric van Blokland wrote:
> Hello everyone,
> 
> A few days back I decided to give Ethernet port bonding in Xen another try.
> I've never been able to get it to work properly and after a short search I
> found the network-bridge-bonding script shipped with CentOS-5 probably
> wasn't going to solve my problems. Instead of searching for a tailored
> solution, I thought it would be a good idea to take a leap and try to
> understand Xen networking a bit better and refresh my networking
> knowledge.
> 
> I came up with a configuration I'm currently testing on a pre-production
> environment and I would love to hear from some experts whether what I'm
> doing is sane or that I'm just plain lucky my ping requests are actually
> coming through. Please don't hesitate to give remarks on efficiency,
> security or questioning my sanity. Especially because networking is
> ordinarily not my piece of cake.
> 
> The hardware setup is simple. We have a physical machine with to gigabit
> Ethernet nics, eth0 and eth1, each connected to its own switch. We want to
> achieve failover by bonding (bond0). We're using mode 0 (round robin) to
> gain some bandwidth as well.
> 
> In this test setup, I'm only running one DomU.
> 
> Now for the configuration. I haven't written a wrapper script for the
> purpose of testing and am just using the ifcfgs.
> 
> First of all I've disabled the bridge script in xend-config:
> (network-script '/bin/true')
> 
> Then setup the interfaces:
> 
> # ifcfg-eth0
> DEVICE=eth0
> USERCTL=no
> BOOTPROTO=none
> MACADDR=00:1E:C9:BB:3B:DE
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> 
> # ifcfg-eth1
> DEVICE=eth1
> USERCTL=no
> BOOTPROTO=none
> MACADDR=00:1E:C9:BB:3B:DF
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> 
> # ifcfg-bond0
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> TYPE=Ethernet
> BONDING_OPTS='mode=0 miimon=100'
> BRIDGE=xenbr0
> 
> # ifcfg-xenbr0
> DEVICE=xenbr0
> BOOTPROTO=none
> ONBOOT=yes
> TYPE=Bridge
> STP=OFF
> FD=9
> HELLO=2
> MAXAGE=12
> 
> # ifcfg-veth0
> DEVICE=veth0
> BOOTPROTO=none
> ONBOOT=yes
> IPADDR=192.168.1.11
> NETMASK=255.255.255.0
> GATEWAY=192.168.1.1
> # Our Dom0 will be needing a HW address
> MACADDR=00:16:3E:00:01:00
> 
> # ifcfg-vif0.0
> DEVICE=vif0.0
> BOOTPROTO=none
> ONBOOT=yes
> BRIDGE=xenbr0
> 
> Whatever firewalling I'm going to do, I presume it would be either on bond0
> and/or veth0. Our DomU's will probably do their own firewalling anyway as
> we can't predict what services they will be running. For the sake of
> efficiency and my lack of iptables knowledge, I've changed some bridge
> settings in systctl.conf:
> 
> # Excerpt sysctl.conf
> net.bridge.bridge-nf-filter-vlan-tagged = 0
> net.bridge.bridge-nf-call-ip6tables = 0
> net.bridge.bridge-nf-call-iptables = 0
> net.bridge.bridge-nf-call-arptables = 0
> 
> Finally it seems the ARP request aren't coming through. I fixed that with
> the following commands:
> 
> ip link set eth0 arp off
> ip link set eth1 arp off
> ip link set bond0 arp off
> ip link set vif0.0 arp off
> ip link set vif1.0 arp off
> 
> A wrapper script could do this for me in the future, as I haven't found an
> ifcfg setting to do this for me. The requirement to disable ARP to achieve
> proper, unrouted bridging, limits the use of certain bonding modes, but I
> don't think I care.
> 
> Ifconfig output after setup:
> 
> bond0     Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           inet6 addr: fe80::21e:c9ff:febb:3bde/64 Scope:Link
>           UP BROADCAST RUNNING NOARP MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:27407897 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:29528747 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:677011912 (645.6 MiB)  TX bytes:2070391875 (1.9 GiB)
> 
> eth0      Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           UP BROADCAST RUNNING NOARP SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:15509051 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14731788 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2338232100 (2.1 GiB)  TX bytes:3153334765 (2.9 GiB)
>           Interrupt:16 Memory:dfdf0000-dfe00000
> 
> eth1      Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           UP BROADCAST RUNNING NOARP SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:11898846 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14796959 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2633747108 (2.4 GiB)  TX bytes:3212024406 (2.9 GiB)
>           Interrupt:17 Memory:dfef0000-dff00000
> 
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:23829 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:23829 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:5148013 (4.9 MiB)  TX bytes:5148013 (4.9 MiB)
> 
> veth0     Link encap:Ethernet  HWaddr 00:16:3E:00:01:00
>           inet addr:192.168.1.11  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::216:3eff:fe00:100/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:27197523 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14952082 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:541462356 (516.3 MiB)  TX bytes:807191362 (769.7 MiB)
> 
> vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
>           UP BROADCAST RUNNING NOARP MULTICAST  MTU:1500  Metric:1
>           RX packets:14952092 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:27197458 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:807192826 (769.7 MiB)  TX bytes:541406194 (516.3 MiB)
> 
> vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
>           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
>           RX packets:197922 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:245822 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:32
>           RX bytes:193048753 (184.1 MiB)  TX bytes:26837000 (25.5 MiB)
> 
> virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
>           inet addr:192.168.122.1  Bcast:192.168.122.255 
> Mask:255.255.255.0 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
> 
> xenbr0    Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           inet6 addr: fe80::21e:c9ff:febb:3bde/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:48495 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:3432718 (3.2 MiB)  TX bytes:398 (398.0 b)
> 
> I hope I haven't forgot a vital setting. Please let me know what you think.
> 
> Kind regards,
> 
> Eric


Eric,

curious to see your progress in this. Up till now I tackled network redundancy 
with multipathing, not with bonding. However, this does not provide added 
speed, though theoretically it should. So I recently decided to switch to 
bonding for the hypervisors in their connections to iSCSI, using rr and 
running over seperate switches, just like you but I'm not at the point of 
installing domU's, so I can't really comment on how and if it works. Will know 
next week though so I will return to this post with my findings...

B.








_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users