WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split

To: Andre Przywara <andre.przywara@xxxxxxx>
Subject: Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split
From: Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
Date: Fri, 28 Jan 2011 07:47:15 +0100
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Delivery-date: Thu, 27 Jan 2011 22:48:05 -0800
Dkim-signature: v=1; a=rsa-sha256; c=simple/simple; d=ts.fujitsu.com; i=juergen.gross@xxxxxxxxxxxxxx; q=dns/txt; s=s1536b; t=1296197237; x=1327733237; h=message-id:date:from:mime-version:to:cc:subject: references:in-reply-to:content-transfer-encoding; z=Message-ID:=20<4D426673.7020200@xxxxxxxxxxxxxx>|Date:=20 Fri,=2028=20Jan=202011=2007:47:15=20+0100|From:=20Juergen =20Gross=20<juergen.gross@xxxxxxxxxxxxxx>|MIME-Version: =201.0|To:=20Andre=20Przywara=20<andre.przywara@xxxxxxx> |CC:=20Keir=20Fraser=20<keir.fraser@xxxxxxxxxxxxx>,=20=0D =0A=20Ian=20Jackson=20<Ian.Jackson@xxxxxxxxxxxxx>,=0D=0A =20"xen-devel@xxxxxxxxxxxxxxxxxxx"=20<xen-devel@xxxxxxxxx source.com>|Subject:=20Re:=20[Xen-devel]=20Hypervisor=20c rash(!)=20on=20xl=20cpupool-numa-split|References:=20<4D4 1FD3A.5090506@xxxxxxx>|In-Reply-To:=20<4D41FD3A.5090506@a md.com>|Content-Transfer-Encoding:=207bit; bh=VGlTsgIThR9pfr+zNmYgqcXGYdCNMcAhCxH81cmDwNc=; b=Q7d33/kRnAwpN/djSLo55cVA0VbUZ4AfTeYuP4bte0qQLFBqNpjDdUSb K6tyMN/mhZLpbVFpE0DYZrt7/8hcv9Vzd6HobhV42fH8anxcymGAvC6ST HSgj5z7s5qhUUzcC4Qx5wgoSc9WCS1O2rBw1FniuHvx0mxu7qMgYRqm5W Iou9E8zcDnhfGl11zHPeARDA9u6Bg0TLPC71zZicAv6d6ewli7ecJjNUL QjKlACMa6paYTK2+j04lZ8n5U7HrE;
Domainkey-signature: s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:Message-ID:Date:From:Organization: User-Agent:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=aMCQ7E9n7DxY7yQEYkPO+uG3vXFIFXY3P1FVhQlYW4E6fvOaGgOTN3et ZKOLdUcqLxSwiNDSnPQGMwpx1wFN2JZOxB+Y132r2tA8ozbL76rlrb+/T YHw6qzkkUDuhP1Wd89+2l2Svq+NOACPONeA6YFckGpGKCqoWW/iwqeASh p7NSNQKODCMOprqt5fqE/aCTqRCY1UH+ShPN6HMGHVr9Jr0lR2rjTuVP3 3oozDDkTJDnt27hglKrF+Hx9leDk2;
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4D41FD3A.5090506@xxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization: Fujitsu Technology Solutions
References: <4D41FD3A.5090506@xxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20101226 Iceowl/1.0b1 Icedove/3.0.11
On 01/28/11 00:18, Andre Przywara wrote:
Hi,

when I boot my machine without restricting Dom0 (dom0_mem=
dom0_max_vcpus=) I get an _hypervisor_ crash when I run
# xl cpupool-numa-split
If Dom0's resources are limited on the Xen cmdline, everything works fine.
The crashdump points to a scheduling problem with weights, so I assume
the NUMA distribution algorithm some fools the hypervisor completely.

I will investigate this further tomorrow, but maybe someone has some
good idea.

I've seen this once with an older cpupool version on a 24 processor machine.
It was NOT related to NUMA, but did occur only on reboot after a Dom0 panic.
The machine had an init script creating a cpupool and populating it with
cpus. The machine was in a panic loop due to the BUG in sched_acct then until
it was resetted manually. After the reset the problem was gone.

As I was never able to reproduce the problem later (the same software is
running on dozens of machines!), I assumed there was a problem related to
the first Dom0 panic, may be some destroyed BIOS tables.

Can the crash be reproduced easily?


Juergen


Regards,
Andre.

root@dosorca:/data/images# xl cpupool-numa-split
(XEN) Xen BUG at sched_credit.c:990
(XEN) ----[ Xen-4.1.0-rc2-pre x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82c4801180f8>] csched_acct+0x11f/0x419
(XEN) RFLAGS: 0000000000010006 CONTEXT: hypervisor
(XEN) rax: 0000000000000010 rbx: 0000000000000f00 rcx: 0000000000000100
(XEN) rdx: 0000000000001000 rsi: ffff830437ffa600 rdi: 0000000000000010
(XEN) rbp: ffff82c480297e10 rsp: ffff82c480297d80 r8: 0000000000000100
(XEN) r9: 0000000000000006 r10: ffff82c4802d4100 r11: 000000afc7df0edf
(XEN) r12: ffff830437ffa5e0 r13: ffff82c480117fd9 r14: ffff830437f9f2e8
(XEN) r15: ffff830434321ec0 cr0: 000000008005003b cr4: 00000000000006f0
(XEN) cr3: 000000080df4e000 cr2: ffff88179af79618
(XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from rsp=ffff82c480297d80:
(XEN) 0000000000000282 fffffed4802d3f80 0000000000000eff ffff830437ffa5e0
(XEN) ffff830437ffa5e8 ffff830437ffa870 ffff830437ffa5e0 0000000000000282
(XEN) ffff830437ffa5e8 00002a3037ffa870 00000f0000000f00 0000000000000000
(XEN) ffff82c400000000 ffff82c4802d3f80 ffff830437ffa5e0 ffff82c480117fd9
(XEN) ffff830437f9f2e8 ffff830437f9f2e0 ffff82c480297e40 ffff82c480125f34
(XEN) 0000000000000002 ffff830437ffa600 ffff82c4802d3f80 000000afb6f8667f
(XEN) ffff82c480297e90 ffff82c480126259 ffff82c48024ae20 ffff82c4802d3f80
(XEN) ffff830437f9f2e0 0000000000000000 0000000000000000 ffff82c4802b0880
(XEN) ffff82c480297f18 ffffffffffffffff ffff82c480297ed0 ffff82c480123327
(XEN) ffff82c4802d4a00 ffff82c480297f18 ffff82c48024ae20 ffff82c480297f18
(XEN) 000000afb6abd652 ffff82c4802d3ec0 ffff82c480297ee0 ffff82c4801233a2
(XEN) ffff82c480297f10 ffff82c4801563f5 0000000000000000 ffff8300c7cd6000
(XEN) 0000000000000000 ffff8300c7ad4000 ffff82c480297d48 0000000000000000
(XEN) 0000000000000000 0000000000000000 ffffffff81a69060 ffff8817a8503f10
(XEN) ffff8817a8503fd8 0000000000000246 ffff8817a8503e80 ffff880000000001
(XEN) 0000000000000000 0000000000000000 ffffffff810093aa 000000aafab2f86e
(XEN) 00000000deadbeef 00000000deadbeef 0000010000000000 ffffffff810093aa
(XEN) 000000000000e033 0000000000000246 ffff8817a8503ef8 000000000000e02b
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 ffff8300c7cd6000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82c4801180f8>] csched_acct+0x11f/0x419
(XEN) [<ffff82c480125f34>] execute_timer+0x4e/0x6c
(XEN) [<ffff82c480126259>] timer_softirq_action+0xf2/0x245
(XEN) [<ffff82c480123327>] __do_softirq+0x88/0x99
(XEN) [<ffff82c4801233a2>] do_softirq+0x6a/0x7a
(XEN) [<ffff82c4801563f5>] idle_loop+0x6a/0x6f
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at sched_credit.c:990
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...




--
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@xxxxxxxxxxxxxx
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel