WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] Random I/O deadlocks in multiple clouds

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] Random I/O deadlocks in multiple clouds
From: Alessandro Grassi <alessandro.grassi@xxxxxxxxx>
Date: Mon, 14 Nov 2011 11:48:37 +0100
Delivery-date: Mon, 14 Nov 2011 02:50:34 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Greetings,

I have two clouds, one running XCP 0.5 and another one running XCP 1.0.

Since a few weeks i'm having problems on both of them:

At one pseudo-random moment one or more of the domUs write this on the
console:

INFO: task [random program] blocked for more than 120 seconds.

The log instead is filled with stacktraces:

http://pastebin.com/ziyyWEXP

>From then on, the VM becomes extremely lagged if not at all unreachable.

The trace suggests it's an I/O problem, but the crashes don't seem to
follow a pattern: they happen during high as well as low I/O traffic,
high/low cpu load, high/low memory usage.

The same thing happens on all dom0s of both my clouds.

The domUs are all running PVOPS enabled kernels (2.6.32+), in a mix of
vanilla+grsec, debian stock and debian backports (lenny/squeeze).

I'm keeping the dom0s under monitoring, but nothing specific seems to
happen during the domU crashes - nothing in xe host-dmesg, nothing in
the graphs.

At this point i'm quite lost, i have no idea how to further debug the
issue.

Does anyone have any suggestions?

Thank you in advance

Sincerely,

--
Alessandro



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>