Re: [Xen-devel] [PATCH] Boot PV guests with more than 128GB (v2) for 3.7

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Wed, Aug 28, 2013 at 08:55:39AM +0100, Jan Beulich wrote:
> >>> On 27.08.13 at 22:34, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> 
> >>> wrote:
> > On Mon, Aug 13, 2012 at 08:54:47AM +0100, Jan Beulich wrote:
> >> >>> On 03.08.12 at 16:46, Konrad Rzeszutek Wilk <konrad@xxxxxxxxxx> wrote:
> >> > Didn't get to it yet. Sorry for top posting. If you have a patch ready I
> >> > can test it on Monday - travelling now.
> >> 
> >> So here's what I was thinking of (compile tested only).
> > 
> > Wow. It took me a whole year to get back to this.
> > 
> > Anyhow I did test it and it worked rather nicely for 64-bit guests. I didn't
> > even try to boot 32-bit guests as the pvops changes I did were only for 
> > 64-bit
> > guests. But if you have a specific kernel for a 32-bit guest I still have
> > the 1TB machine for a week and can boot it up there.
> 
> Considering that you had also attached a debug patch - did it
> work without that, i.e. just with the patch that I had handed
> you? If so, I'd then finally be in the position to submit this,
> putting your Tested-by (and perhaps Reported-by) underneath.

Yes it did with the 'memory=440000' guest config. I developed the
debug patch just to make sure I could see the failing case (fix=0) and
working case (fix=1) without having to reboot this monster machine.


Interestingly enough if I boot with a 486GB guest I end up with:

[root@ca-test111 konrad]# xl dmesg | tail -300
(XEN) d8:v0: unhandled page fault (ec=0000)
(XEN) Pagetable walk from ffff880043e75070:
(XEN)  L4[0x110] = 00000080ba854067 0000000000001a0d
(XEN)  L3[0x001] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 8 (vcpu#0) crashed on cpu#16:
(XEN) ----[ Xen-4.4-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    16
(XEN) RIP:    e033:[<ffffffff81acd29e>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: ffffffff8219e000
(XEN) rdx: 0000000000000000   rsi: ffff880043e75000   rdi: 00000000deadbeef
(XEN) rbp: ffffffff81a01ff8   rsp: ffffffff81a01f00   r8:  0000000043e7a000
(XEN) r9:  0000000043e7b000   r10: 0000000000223000   r11: 000000a0a66b6067
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000400fcb6000   cr2: ffff880043e75070
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81a01f00:
(XEN)    ffffffff8219e000 000000a0a66b6067 0000000000000000 ffffffff81acd29e
(XEN)    000000010000e030 0000000000010046 ffffffff81a01f48 000000000000e02b
(XEN)    ffffffff81acd267 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 809822011f898975
(XEN)    000206e501200800 0000000000000001 0000000000000000 0000000000000000
(XEN)    0f00000060c0c748 ccccccccccccc305 cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc
(XEN)    cccccccccccccccc cccccccccccccccc cccccccccccccccc cccccccccccccccc

(this is with the debug patch and the guest having 'fix=1' enabled, meaning
it uses the new code path).

Thought looking at the stack more, I see:

ffffffff81acd29e is:

   0xffffffff81acd280 <xen_start_kernel+935>:   mov    $0xffffffff81931558,%rdi
   0xffffffff81acd287 <xen_start_kernel+942>:   xor    %eax,%eax
   0xffffffff81acd289 <xen_start_kernel+944>:   callq  0xffffffff813f5340 
<xen_raw_printk>
   0xffffffff81acd28e <xen_start_kernel+949>:   mov    0x1a6f53(%rip),%rsi      
  # 0xffffffff81c741e8 <xen_start_info>
   0xffffffff81acd295 <xen_start_kernel+956>:   movb   $0x90,0x1aa454(%rip)     
   # 0xffffffff81c776f0 <boot_params+528>
   0xffffffff81acd29c <xen_start_kernel+963>:   xor    %edx,%edx
   0xffffffff81acd29e <xen_start_kernel+965>:   mov    0x70(%rsi),%rax

which implies that we copied from the 

  xen_start_info something (pt_base? mod_start?) which has the __va

address instead of the __kva one. So the bootup pagetables creation
I think we are OK with and indeed you can put 'Tested-by' tag on it.

I will dig in this a bit more.
> 
> And no, I'm not really concerned about the 32-bit case. The
> analogy with the 64-bit code is sufficient to tell that the change
> (even if just cosmetic) should also be done to the 32-bit variant.

Right.
> 
> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel