WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: Re; [Xen-users] Ethernet has Alzheimers

To: Matthew Crocker <mcrocker@xxxxxxxxxxx>
Subject: Re: Re; [Xen-users] Ethernet has Alzheimers
From: Gareth Bult <gareth@xxxxxxxxxxxxx>
Date: Tue, 8 Jan 2008 00:39:09 +0000 (GMT)
Cc: xen-users@xxxxxxxxxxxxxxxxxxx, Gareth Bult <gareth@xxxxxxxxxxxxx>
Delivery-date: Mon, 07 Jan 2008 16:42:07 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <8F9EEEAF-C7EC-4FE3-8113-E097C787C320@xxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
The traffic does not leave my machine ...

Here's my ifconfig if that helps;

0         Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:15:C5:5D:C0:BE  
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::215:c5ff:fe5d:c0be/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:176328 errors:0 dropped:0 overruns:0 frame:0
          TX packets:301583 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:26003737 (24.7 MB)  TX bytes:488759515 (466.1 MB)

eth1      Link encap:Ethernet  HWaddr 00:15:C5:5D:C0:BF  
          inet addr:87.102.101.158  Bcast:87.102.101.255  Mask:255.255.255.0
          inet6 addr: fe80::215:c5ff:fe5d:c0bf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:272976 errors:0 dropped:0 overruns:0 frame:0
          TX packets:559429 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:17198305 (16.4 MB)  TX bytes:249087754 (237.5 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:79887 errors:0 dropped:0 overruns:0 frame:0
          TX packets:79887 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6587300 (6.2 MB)  TX bytes:6587300 (6.2 MB)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:13260 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20341 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1038838 (1014.4 KB)  TX bytes:5661953 (5.3 MB)
          Base address:0xecc0 Memory:fe9e0000-fea00000 

peth1     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:846060 errors:0 dropped:0 overruns:0 frame:0
          TX packets:577736 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:267309371 (254.9 MB)  TX bytes:260521447 (248.4 MB)
          Base address:0xdcc0 Memory:fe5e0000-fe600000 

tun0      Link encap:UNSPEC  HWaddr 
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.99.0.1  P-t-P:10.99.0.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:301583 errors:0 dropped:0 overruns:0 frame:0
          TX packets:176329 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:488759515 (466.1 MB)  TX bytes:26003807 (24.7 MB)

vif0.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:559429 errors:0 dropped:0 overruns:0 frame:0
          TX packets:272977 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:249087754 (237.5 MB)  TX bytes:17198375 (16.4 MB)

vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:164166 errors:0 dropped:0 overruns:0 frame:0
          TX packets:182474 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:22467019 (21.4 MB)  TX bytes:305137759 (291.0 MB)

vif4.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:7535 errors:0 dropped:0 overruns:0 frame:0
          TX packets:276712 errors:0 dropped:2052 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:4698540 (4.4 MB)  TX bytes:23528102 (22.4 MB)

vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:99546 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108154 errors:0 dropped:9 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:13948995 (13.3 MB)  TX bytes:178514052 (170.2 MB)

vif5.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:4779 errors:0 dropped:0 overruns:0 frame:0
          TX packets:271874 errors:0 dropped:2778 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:2391054 (2.2 MB)  TX bytes:21580528 (20.5 MB)

vif6.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:6701 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3462 errors:0 dropped:14 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:430729 (420.6 KB)  TX bytes:657222 (641.8 KB)

vif6.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:806 errors:0 dropped:0 overruns:0 frame:0
          TX packets:265698 errors:0 dropped:2098 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:123889 (120.9 KB)  TX bytes:19099185 (18.2 MB)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:98256 errors:0 dropped:0 overruns:0 frame:0
          TX packets:183 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:13375927 (12.7 MB)  TX bytes:7770 (7.5 KB)

xenbr1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:825898 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2447 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:249201663 (237.6 MB)  TX bytes:102934 (100.5 KB)


--
Managing Director, Encryptec Limited
Tel: 0845 25 77033, Mob: 07853 305393, Int: 00 44 1443205756
Email: gareth@xxxxxxxxxxxxx 
Statements made are at all times subject to Encryptec's Terms and Conditions of 
Business, which are available upon request.

----- Original Message -----
From: "Matthew Crocker" <mcrocker@xxxxxxxxxxx>
To: "Gareth Bult" <gareth@xxxxxxxxxxxxx>
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Sent: Tuesday, January 8, 2008 12:24:23 AM (GMT) Europe/London
Subject: Re: Re; [Xen-users] Ethernet has Alzheimers

> Wait, it gets better, on 10.0.0.12;
>
> arp -d 10.0.0.1
>
> tcpdump shows;
>
> 23:14:14.014725 arp who-has 10.0.0.1 tell 10.0.0.10
> 23:14:14.014797 arp reply 10.0.0.1 is-at fe:ff:ff:ff:ff:ff

Look at the switch and fine out what port has FE:FF:FF:FF:FF:FF on it  
and hit that machine in the head.

-Matt

> Eeeek!
>
> --
> Managing Director, Encryptec Limited
> Tel: 0845 25 77033, Mob: 07853 305393, Int: 00 44 1443205756
> Email: gareth@xxxxxxxxxxxxx
> Statements made are at all times subject to Encryptec's Terms and  
> Conditions of Business, which are available upon request.
>
> ----- Original Message -----
> From: "Gareth Bult" <gareth@xxxxxxxxxxxxx>
> To: "Matthew Crocker" <mcrocker@xxxxxxxxxxx>
> Cc: xen-users@xxxxxxxxxxxxxxxxxxx
> Sent: Monday, January 7, 2008 11:11:04 PM (GMT) Europe/London
> Subject: Re: Re; [Xen-users] Ethernet has Alzheimers
>
> Ok, I've managed to pin it down and you are quite right - it's ARP.
>
> Now the question is, how do I fix it.
>
> here's what I have
>
> Dom0 :: 10.0.0.1
> DomU :: 10.0.0.12
>
> Both machines work fine for 40 mins .. then;
>
> DomU reports Dom0 unreachable.
> Sure enough ping 10.0.0.1 gives no response.
> However, ping 10.0.0.12 from Dom0 responds fine.
> A one-way ping!
>
> arp -na on Dom0 reports as expected.
> arp -na on the broken DomU shows;
> ? (10.0.0.1) at FE:FF:FF:FF:FF:FF [ether] on eth0
>
> It's picking up FE:EE ... instead of the desired MAC address ?!
>
> How can it do this ?!
>
> On Dom0:
> eth0      Link encap:Ethernet  HWaddr 00:15:C5:5D:C0:BE
>          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
>          inet6 addr: fe80::215:c5ff:fe5d:c0be/64 Scope:Link
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:205397 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:413848 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:0
>          RX bytes:41267633 (39.3 MB)  TX bytes:95050228 (90.6 MB)
>
> On DomU:
> eth0      Link encap:Ethernet  HWaddr 00:00:10:00:00:0C
>          inet addr:10.0.0.12  Bcast:10.0.0.255  Mask:255.255.255.0
>          inet6 addr: fe80::200:10ff:fe00:c/64 Scope:Link
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:6297 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:11662 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:1000
>          RX bytes:1157351 (1.1 MB)  TX bytes:907972 (886.6 KB)
>
> Help!
>
>
>
>
> --
> Managing Director, Encryptec Limited
> Tel: 0845 25 77033, Mob: 07853 305393, Int: 00 44 1443205756
> Email: gareth@xxxxxxxxxxxxx
> Statements made are at all times subject to Encryptec's Terms and  
> Conditions of Business, which are available upon request.
>
> ----- Original Message -----
> From: "Matthew Crocker" <mcrocker@xxxxxxxxxxx>
> To: "Gareth Bult" <gareth@xxxxxxxxxxxxx>
> Cc: xen-users@xxxxxxxxxxxxxxxxxxx
> Sent: Monday, January 7, 2008 10:00:40 PM (GMT) Europe/London
> Subject: Re: Re; [Xen-users] Ethernet has Alzheimers
>
>>> Is the machine answering ARP replies?
>>
>> Honestly, I don't know .. the machine tends to lock up for other
>> reasons when it dies hence it's not easy to track ..
>>
>>> Does the upstream router have the IP & MAC in its ARP table?
>>> Does the upstream switch have the MAC in its mac-address-table?
>>> Assigned to the correct port?
>>
>>> Sounds to me like an ARP timeout problem.
>>
>> This occurs between DomU's and Dom0 in addition to external
>> addresses ... so I don't think it's linked to anything outside of
>> Xen .. I've experienced the same problem on 4 different machines,
>> all different HW config .. so again I think faulty HW is out.
>>
>> For what it's worth;
>>
>> I'm using Ubuntu Gutsy (7.10) with the stock Xen 3.1 kernel all on
>> AMD64 and Intel/Xeon machines all running 64 bit kernels and distros.
>>
>> All machines are using bridging with two physical ethernet ports.
>> All DomU's are running two matching virtual ports.
>> I'm using IPTABLES (firehol) fairly heavily for port filtering.
>
> Bridging is Layer2,  IP is Layer 3, you are having a problem at layer
> 3 so you need to look to make sure your layer 2 stuff is working
> properly.
>
> If Xen is bridging only then you won't really have visibility into the
> Layer 3 problem from Dom0.  You could look at the bridging config and
> see if it knows about the MAC address properly in the switch.   At
> some point upstream from the Xen hardware you have another Layer 3
> device,  most likely a router.  You need to get into that router and
> see if it has the IP -> MAC entry in its ARP table.  If it doesn't
> have it then there is your problem.  Something is stopping the DomU
> from answering the ARP queries from the router.  The route loses track
> of the MAC address and can no longer send Ethernet frames to your
> DomU.  If your router does have the ARP entry then I would look into
> your switches and see if they are dropping the MAC address from their
> table.
>
>
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users