Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

[Hanweidong <hanweidong@xxxxxxxxxx>]

> -----Original Message-----
> From: xen-devel-bounces@xxxxxxxxxxxxx [mailto:xen-devel-
> bounces@xxxxxxxxxxxxx] On Behalf Of Hanweidong
> Sent: 2013年3月26日 17:38
> To: Stefano Stabellini
> Cc: George Dunlap; xudong.hao@xxxxxxxxx; Yanqiangjun; Luonengjun;
> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> devel@xxxxxxxxxxxxx; xiantao.zhang@xxxxxxxxx
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> 
> > -----Original Message-----
> > From: Stefano Stabellini [mailto:stefano.stabellini@xxxxxxxxxxxxx]
> > Sent: 2013年3月18日 20:02
> > To: Hanweidong
> > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > devel@xxxxxxxxxxxxx; xudong.hao@xxxxxxxxx; xiantao.zhang@xxxxxxxxx
> > Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> > with 4G memory
> >
> > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
> > code to init
> > > RAM in xen_ram_init:
> > >
> > >     ...
> > >     block_len = ram_size;
> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > >         /* Xen does not allocate the memory continuously, and keep
> a
> > hole at
> > >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> > >          */
> > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > >     }
> > >     memory_region_init_ram(&ram_memory, "xen.ram", block_len);
> > >     vmstate_register_ram_global(&ram_memory);
> > >
> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > >     } else {
> > >         below_4g_mem_size = ram_size;
> > >     }
> > >     ...
> > >
> > > HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> > to e0000000,
> > > Which it's consistent with hvmloader when assigning a GPU, and then
> > guest worked
> > > for us. So we wondering that xen_ram_init in QEMU should be
> > consistent with
> > > hvmloader.
> > >
> > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
> as
> > below.
> > > Should keep these places handle the consistent mmio hole or not?
> > >
> > >     if (ram_size >= 0xe0000000 ) {
> > >         above_4g_mem_size = ram_size - 0xe0000000;
> > >         below_4g_mem_size = 0xe0000000;
> > >     } else {
> > >         above_4g_mem_size = 0;
> > >         below_4g_mem_size = ram_size;
> > >     }
> >
> > The guys at Intel sent a couple of patches recently to fix this issue:
> >
> > http://marc.info/?l=xen-devel&m=136150317011027
> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> >
> > Do they solve your problem?
> 
> These two patches didn't solve our problem.
> 

I debugged this issue with above two patches. I want to share some information 
and discuss solution here. This issue is actually caused by that a VM has a 
large pci hole (mmio size) which results in QEMU sets memory regions 
inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1 and 
xen_ram_init). I created a virtual device with 1GB mmio size to debug this 
issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() 
and xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and 
wrote it to TOM register, which triggered QEMU to update pci hole region with 
0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM 
(configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU 
pc_init1 and xen_ram_init to match hvmloader's. Then the problem was gone. 

Althrough above two patches will pass actual pci hole start address to QEMU, 
but it's too late, QEMU pc_init1() and xen_ram_init() already set the other 
memory regions, and obviously the pci hole might overlap with ram regions in 
this case. So I think hvmloader should setup pci devices and calculate pci hole 
first, then QEMU can map memory regions correctly from the beginning.  

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel