WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] kernel oops/IRQ exception when networking betweenmany do

To: Ian Pratt <m+Ian.Pratt@xxxxxxxxxxxx>
Subject: RE: [Xen-devel] kernel oops/IRQ exception when networking betweenmany domUs
From: Nils Toedtmann <xen-devel@xxxxxxxxxxxxxxxxxx>
Date: Wed, 08 Jun 2005 17:22:52 +0200
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Wed, 08 Jun 2005 15:22:09 +0000
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <A95E2296287EAD4EB592B5DEEFCE0E9D282120@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <A95E2296287EAD4EB592B5DEEFCE0E9D282120@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Am Mittwoch, den 08.06.2005, 16:01 +0100 schrieb Ian Pratt:
> You might want to compile the kernel with CONFIG_FRAME_POINTERS to get more 
> accurate call traces. 
> 
> I'd also add some printk's to the 'unusual' paths in netback.c:net_rx_action, 
> such as the memory squeeze path.

? (have no clue about kernel programming. I'm only good for panicking
kernels ;) 

> It's also worth building Xen with debug=y and connect a serial line.

That'll be some work ... i'll do so if there is interest in this bug and
if somebody else can confirm that "ping -f -b $BROADCAST_IP" on xen-br0
kills dom0 when there are enough domUs.

/nils.

> Best,
> Ian
> > -----Original Message-----
> > From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx 
> > [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
> > Nils Toedtmann
> > Sent: 08 June 2005 15:40
> > To: Birger Tödtmann
> > Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
> > Subject: Re: [Xen-devel] kernel oops/IRQ exception when 
> > networking betweenmany domUs
> > 
> > Am Mittwoch, den 08.06.2005, 14:34 +0200 schrieb Nils Toedtmann:
> > [...] 
> > > Ok, reproduced the dom0 kernel panic in a simpler situation:
> > > 
> > > * create some domUs, each having 1 interface in the same subnet
> > > * bridge all the interfaces together (dom0 not having an ip on that
> > >   bridge)
> > > * trigger unicast traffic as much as you want (like unicast flood
> > >   pings): No problem.
> > > * Now trigger some broadcast traffic between the domUs:
> > > 
> > >     ping -i 0,1 -b 192.168.0.255
> > > 
> > >   BOOOM.
> > > 
> > > 
> > > Instead, you may down all vifs first, start the flood 
> > broadcast ping 
> > > in the first domU and bring up one vif after the other 
> > (wait each time
> > > >15sec until the bridge put the added port in forwarding 
> > state). After
> > > bringing up 10-15 vifs, dom0 panics. 
> > > 
> > > I could _not_ reproduce this with massive unicast traffic. 
> > The problem 
> > > disappears if i set "net.ipv4.icmp_echo_ignore_broadcasts=1" in all 
> > > domains. Maybe the probem rises if to many domUs answer to 
> > broadcasts 
> > > at the same time (collisions?).
> > 
> > 
> > More testing: again doing a 
> > 
> >   [root@domUtest01 ~]# ping -f -b 192.168.0.255
> > 
> > into the bridged vif-subnet. With all domains having 
> > "net.ipv4.icmp_echo_ignore_broadcasts=1" (so noone answers 
> > the pings) everything is fine. When i switch in the pinging 
> > domUtest01 itself (and _only_ in that domain) to 
> > "net.ipv4.icmp_echo_ignore_broadcasts=0", dom0 immediately 
> > panics (if there are 15-20 domUs in that bridged subnet).
> > 
> > Another test: putting dom0's vif0.0 on the bridge too, 
> > pinging from dom0. Then in needed (yet) all domains to have 
> > "net.ipv4.icmp_echo_ignore_broadcasts=0" to get my oops.
> > 
> > The oopses happen in different places, not all contain 
> > "net_rx_action" (all are "Fatal exception in interupt". These "dumps"
> > may contain typos because i copied them from monitor by hand):
> > 
> >   [...]
> >   error_code
> >   kfree_skbmem
> >   __kfree_skb
> >   net_rx_action
> >   tasklet_action
> >   __do_softirq
> >   soft_irq
> >   irq_exit
> >   do_IRQ
> >   evtchn_do_upcall
> >   hypervisor_callback
> >   __wake_up
> >   sock_def_readable
> >   unix_stream_sendmsg
> >   sys_sendto
> >   sys_send
> >   sys_socketcall
> >   syscall_call
> > 
> > or
> > 
> >   [...]
> >   error_code
> >   tasklet_action
> >   __do_softirq
> >   soft_irq
> >   irq_exit
> >   do_IRQ
> >   evtchn_do_upcall
> >   hypervisor_callback
> > 
> > or
> > 
> >   [...]
> >   error_code
> >   tasklet_action
> >   __do_softirq
> >   soft_irq
> >   evtchn_do_upcall
> >   hypervisor_callback
> >   cpu_idle
> >   start_kernel
> > 
> > or
> > 
> >   [...]
> >   error_code
> >   kfree_skbmem
> >   __kfree_skb
> >   net_rx_action
> >   tasklet_action
> >   __do_softirq
> >   soft_irq
> >   irq_exit
> >   do_IRQ
> >   evtchn_do_upcall
> >   hypervisor_callback
> >   __mmx_memcpy
> >   memcpy
> >   dup_task_struct
> >   copy_process
> >   do_fork
> >   sys_clone
> >   syscall_call
> > 
> > or
> > 
> >   [...]
> >   error_code
> >   kfree_skbmem
> >   __kfree_skb
> >   net_rx_action
> >   tasklet_action
> >   __do_softirq
> >   soft_irq
> >   irq_exit
> >   do_IRQ
> >   evtchn_do_upcall
> >   hypervisor_callback
> >   __wake_up
> >   sock_def_readable
> >   unix_stream_sendmsg
> >   sys_sendto
> >   sys_send
> >   sys_socketcall
> >   syscall_call
> > 
> > or
> > 
> >   [...]
> >   error_code
> >   kfree_skbmem
> >   __kfree_skb
> >   net_rx_action
> >   tasklet_action
> >   __do_softirq
> >   do_softirq
> >   local_bh_enable
> >   dev_queue_xmit
> >   nf_hook_slow
> >   ip_finish_output
> >   dst_output
> >   ip_push_pending_frames
> >   raw_sendmsg
> >   sock_sendmsg
> >   sys_sendmsg
> >   sys_socketcall
> >   syscall_call
> > 
> > and more ...
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> > 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>
  • RE: [Xen-devel] kernel oops/IRQ exception when networking betweenmany domUs, Nils Toedtmann <=