WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] bring up Hypervisor on large (512GB) memory

To: "'mukesh.rathor@xxxxxxxxxx'" <mukesh.rathor@xxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Subject: RE: [Xen-devel] bring up Hypervisor on large (512GB) memory
From: "Tian, Kevin" <kevin.tian@xxxxxxxxx>
Date: Tue, 10 Feb 2009 12:52:04 +0800
Accept-language: en-US
Acceptlanguage: en-US
Cc:
Delivery-date: Mon, 09 Feb 2009 20:52:36 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4990EFC7.9000907@xxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4990EFC7.9000907@xxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AcmLLQGX/K+S8QgSQvqnh1XIvpe+eAAC949A
Thread-topic: [Xen-devel] bring up Hypervisor on large (512GB) memory
>From: Mukesh Rathor
>Sent: Tuesday, February 10, 2009 11:09 AM
>
>Hi all,
>
>Trying to bring up the hypervisor on 512GB :
>
>1. Started with xen 3.1.4 (last oracle release), and 512GB, the system
>panic'd right away:
>    (XEN) Early fatal page fault at e008:ffff828c801ce0ad
>    (cr2=ffff8300cfc00000, ec=0002)
>
>I noticed an anomaly here in the RAM map:
>(XEN) Xen-e820 RAM map:
>(XEN)  0000000000000000 - 000000000009f400 (usable)
>(XEN)  000000000009f400 - 00000000000a0000 (reserved)
>(XEN)  00000000000f0000 - 0000000000100000 (reserved)
>(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
>(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
>(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
>(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
>(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
>(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
>(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
>(XEN)  0000000100000000 - 000000802ffff000 (usable)  <-------
>
>Seems that should be 0000008000000000 ???

802ffff000 seems more reasonable since there's a big hole reserved 
under 4G and thus max page in address space is shifted up to 
exceed 512GB size

>
>
>2. Reduce some RAM, and booting with 440GB, map makes more sense:
>
>(XEN) Xen-e820 RAM map:
>(XEN)  0000000000000000 - 000000000009f400 (usable)
>(XEN)  000000000009f400 - 00000000000a0000 (reserved)
>(XEN)  00000000000f0000 - 0000000000100000 (reserved)
>(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
>(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
>(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
>(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
>(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
>(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
>(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
>(XEN)  0000000100000000 - 0000006e00000000 (usable)
>

When you say "reducing some RAM", did you plug off DIMMs 
physically or just use "mem=" option to have Xen ignore pages
beyond that size? If the latter, then it makes sense to observe
6e00000000.

>Panic again:
>(XEN) Early fatal page fault at e008:ffff828c80210460
>(cr2=ffff8300cfc00000, ec=0002)
>
>
>The panic is trying to allocate bitmap in 
>init_boot_allocator(). The bitmap
>starts at cfac0000 and given the size dc1000, won't fit.
>
>
>3. Moving to unstable 19164, looks like things are even more 
>tighter!! I
>    couldn't even boot with 64GB. bitmap_start:cfac0000, 
>bitmap_size:201000,
>    alloc_bitmap:ffff8300cfac0000
>
>
>The only solution I can think of is moving the bitmap 
>elsewhere, above 4GB in
>this case:
>    figure the size of bitmap, DIRECT map space, allocate the map,
>    mark it reserved in the RAM map, and should work!
>
>    I'd have add a loop around init_boot_allocator() in __start_xen()
>    iterating thru the RAM map again, and finding space above 16M.
>
>
>Am I on the right track?
>

that bitmap follows xen image immediately, and there's a limitation
to only choose e820 entry (< BOOTSTRAP_DIRECTMAP_END) for
Xen relocation. I don't know the tricks behind that limitation. But if
you want to move bitmap higher, is it more generic to allow Xen image
itself relocated to >4G trunk which also solves your case here? Of
course I may lose some background here...

Or alternative is to have reloc_size covering bitmap size with smaller
change.

Thanks,
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel