Re: [Xen-devel] Hyp compat_memory_op() and 256 GB PV

[Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>]

Hi Keir,

Attached please find my patch to allocate 1M pages at a time. With this patch 
on 3.3.1 and unstable, I'm able to bring up the hypervisor with 512GB memory 
and start a PV 64 guest (with my pud entry fix).
The only issue is performance related. As somewhat expected, after guest 
shutdown it takes a while for pages to be freed and guest to disappear. Until 
this time, system is fine, but xm will hang.
In my kdb debugger, I see cpu's in :
[0]ffff828c80110f52: page_scrub_softirq+22          test %al, %al
[1]ffff828c80110f52: page_scrub_softirq+22          test %al, %al
[2]ffff828c80110f52: page_scrub_softirq+22          test %al, %al
...

page_scrub_softirq():
    /* free_heap_pages() does not parallelise well. Serialise this function. */
    if ( !spin_trylock(&serialise_lock) )

I guess a future work item to look into parallelizing this some day....

Thanks a lot,
Mukesh



Keir Fraser wrote:
> On 18/02/2009 19:52, "Mukesh Rathor" <mukesh.rathor@xxxxxxxxxx> wrote:
>
>> This is a PV domain BTW, I guess xc_hvm_build.c wont' be involved. I'll
>> instrument libxc and debug more... meanwhile I thought I'd post this.
>
> Yes, my mistake, and this points us straight at the problem.
>
> You see the xc_domain_memory_populate_physmap() call at the end of
> libxc/xc_dom_x86.c:arch_setup_meminit()? That needs to be put in a for loop,
> allocating no more than, say, a million pages at a time.
>
> The patch should be very simple, so I'll leave the rest to you.
>
>  -- Keir
>

Signed-off-by: Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>
---
diff -r 746955812d23 tools/libxc/xc_dom_x86.c
--- a/tools/libxc/xc_dom_x86.c  Wed Mar 11 17:47:56 2009 -0700
+++ b/tools/libxc/xc_dom_x86.c  Wed Mar 18 14:31:07 2009 -0700
@@ -25,6 +25,7 @@
 #include "xenctrl.h"
 
 /* ------------------------------------------------------------------------ */
+#define MEM_CHUNKSZ                  1024*1024
 
 #define bits_to_mask(bits)       (((xen_vaddr_t)1 << (bits))-1)
 #define round_down(addr, mask)   ((addr) & ~(mask))
@@ -694,7 +695,7 @@ int arch_setup_meminit(struct xc_dom_ima
 int arch_setup_meminit(struct xc_dom_image *dom)
 {
     int rc;
-    xen_pfn_t pfn;
+    xen_pfn_t pfn, pages, allocsz, *p2mp;
 
     rc = x86_compat(dom->guest_xc, dom->guest_domid, dom->guest_type);
     if ( rc )
@@ -712,10 +713,16 @@ int arch_setup_meminit(struct xc_dom_ima
     for ( pfn = 0; pfn < dom->total_pages; pfn++ )
         dom->p2m_host[pfn] = pfn;
 
-    /* allocate guest memory */
-    rc = xc_domain_memory_populate_physmap(dom->guest_xc, dom->guest_domid,
-                                           dom->total_pages, 0, 0,
-                                           dom->p2m_host);
+    /* allocate guest memory MEM_CHUNKSZ at a time */
+    for ( rc=0, p2mp=dom->p2m_host, pages=dom->total_pages; !rc && pages; )
+    {
+        allocsz = (pages <= MEM_CHUNKSZ) ? pages : MEM_CHUNKSZ;
+        rc = xc_domain_memory_populate_physmap(dom->guest_xc, dom->guest_domid,
+                                               allocsz, 0, 0, p2mp);
+        pages -= allocsz;
+        p2mp += allocsz;
+    }
+
     return rc;
 }
 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel