WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-api

Re: [Xen-API] Flood test with xen/openvswitch

To: "Andres E. Moya" <amoya@xxxxxxxxxxxxxxxxx>
Subject: Re: [Xen-API] Flood test with xen/openvswitch
From: George Shuklin <george.shuklin@xxxxxxxxx>
Date: Wed, 07 Sep 2011 19:04:27 +0400
Cc: xen-api@xxxxxxxxxxxxxxxxxxx
Delivery-date: Wed, 07 Sep 2011 08:11:45 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:from:to:cc:in-reply-to:references:content-type:date :message-id:mime-version:x-mailer:content-transfer-encoding; bh=U31eUe8YWu0sPDlMY7IPH7rFV3x88hgOelNY89NwJ9Q=; b=loBrToL3iNQTxxBIGFLe4leBxs7BoNGhUvD6TVD4w1K0PnbA+VTB8ZjaUlgFOtHMLD 4/VB1vKOqOkbqSo1uI2r0sYNNWbiq7KQkRqx2IxLcN/z+cBwSCyrkdFagV3impzy+0Uu dxP1IRQC8mZc0BHjGIVzOZj1ojBd433QJBe00=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <88e9eb6c-4c7a-44d6-9460-5ff7e87bf18c@mbs01>
List-help: <mailto:xen-api-request@lists.xensource.com?subject=help>
List-id: Discussion of API issues surrounding Xen <xen-api.lists.xensource.com>
List-post: <mailto:xen-api@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-api>, <mailto:xen-api-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-api>, <mailto:xen-api-request@lists.xensource.com?subject=unsubscribe>
References: <88e9eb6c-4c7a-44d6-9460-5ff7e87bf18c@mbs01>
Sender: xen-api-bounces@xxxxxxxxxxxxxxxxxxx
I think this can be real issue.

We have a bunch of highloaded hosts with multiple CPUs in dom0 (used to
reduce network latency in load peaks).

So I'm thinking issue can be not with openvswitch, but with
hardware/compability with xen...

В Ср., 07/09/2011 в 10:43 -0400, Andres E. Moya пишет:
> wouldn't this give us the crashing issue that has been occurring in xen?
> 
> recntly i had to run this command
> echo "NR_DOMAIN0_VCPUS=1" > /etc/sysconfig/unplug-vcpus to stop xen from 
> crashing, its been running for 24 hours now.
> 
> Moya Solutions, Inc.
> amoya@xxxxxxxxxxxxxxxxx
> 0 | 646-918-5238 x 102
> F | 646-390-1806
> 
> ----- Original Message -----
> From: "George Shuklin" <george.shuklin@xxxxxxxxx>
> To: xen-api@xxxxxxxxxxxxxxxxxxx
> Sent: Wednesday, September 7, 2011 9:59:13 AM
> Subject: Re: [Xen-API] Flood test with xen/openvswitch
> 
> temporary solution: add more active cpus to dom0.
> echo 1 >/sys/devices/system/cpu/cpu1/online
> echo 1 >/sys/devices/system/cpu/cpu2/online
> echo 1 >/sys/devices/system/cpu/cpu3/online
> 
> В Ср., 07/09/2011 в 13:29 +0200, Sébastien Riccio пишет:
> > Hi,
> > 
> > I just did a test to see how openvswitch handle a flood from a virtual 
> > machine on a xen
> > host using it as the networking layer.
> > 
> > I just issued a :
> > 
> > vm1# hping3  -S -L 0 -p 80 -i u100 192.168.1.1
> > 
> > options I used are:
> > -S set SYN tcp flag
> > -L set ACK tcp flag
> > -p destination port
> > -i u100 = interval between packets in micro seconds
> > 
> > This results in a cpu usage up to 97% by the ovs-vswitchd process in the 
> > dom0.
> > Letting it go for a few minutes turns the whole xen host unresponsive to 
> > network access
> > and must then be accessed from the local console.
> > 
> > Is that an excepted behavior ? I know the test is quite agressive but 
> > any customer could issue such a flood
> > and render the whole host unreachable. Are there workarounds ?
> > 
> > Thanks for your help.
> > 
> > Best regards,
> > Sébastien
> > 
> > part of the openvswitch logs when issuing the flood:
> > 
> > Sep 07 13:13:26|01523|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (85% 
> > CPU usage)
> > Sep 07 13:13:26|01524|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (85% 
> > CPU usage)
> > Sep 07 13:13:27|01525|poll_loop|WARN|Dropped 5136 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:27|01526|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (85% 
> > CPU usage)
> > Sep 07 13:13:27|01527|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (85% 
> > CPU usage)
> > Sep 07 13:13:28|01528|poll_loop|WARN|Dropped 5815 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:28|01529|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (85% 
> > CPU usage)
> > Sep 07 13:13:28|01530|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (85% 
> > CPU usage)
> > Sep 07 13:13:29|01531|poll_loop|WARN|Dropped 8214 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:29|01532|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (82% 
> > CPU usage)
> > Sep 07 13:13:29|01533|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (82% 
> > CPU usage)
> > Sep 07 13:13:30|01534|poll_loop|WARN|Dropped 5068 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:30|01535|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (82% 
> > CPU usage)
> > Sep 07 13:13:30|01536|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (82% 
> > CPU usage)
> > Sep 07 13:13:31|01537|poll_loop|WARN|Dropped 5008 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:31|01538|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (82% 
> > CPU usage)
> > Sep 07 13:13:31|01539|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (82% 
> > CPU usage)
> > Sep 07 13:13:32|01540|poll_loop|WARN|Dropped 4841 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:32|01541|poll_loop|WARN|wakeup due to 40-ms timeout at 
> > ../ofproto/ofproto-dpif.c:622 (83% CPU usage)
> > Sep 07 13:13:32|01542|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (83% 
> > CPU usage)
> > Sep 07 13:13:33|01543|poll_loop|WARN|Dropped 92 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:33|01544|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (83% 
> > CPU usage)
> > Sep 07 13:13:33|01545|poll_loop|WARN|wakeup due to [POLLIN] on fd 21 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (83% 
> > CPU usage)
> > Sep 07 13:13:34|01546|poll_loop|WARN|Dropped 27 log messages in last 1 
> > seconds (most recently, 1 seconds ago) due to excessive rate
> > Sep 07 13:13:34|01547|poll_loop|WARN|wakeup due to 53-ms timeout at 
> > ../lib/mac-learning.c:294 (83% CPU usage)
> > Sep 07 13:13:34|01548|poll_loop|WARN|wakeup due to [POLLIN] on fd 18 
> > (NETLINK_GENERIC<->NETLINK_GENERIC) at ../lib/netlink-socket.c:668 (83% 
> > CPU usage)
> > 
> > 
> > _______________________________________________
> > xen-api mailing list
> > xen-api@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/mailman/listinfo/xen-api
> 
> 
> 
> _______________________________________________
> xen-api mailing list
> xen-api@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/mailman/listinfo/xen-api
> 
> 



_______________________________________________
xen-api mailing list
xen-api@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/mailman/listinfo/xen-api