Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

[George Dunlap <George.Dunlap@xxxxxxxxxxxxx>]

On Thu, Apr 25, 2013 at 4:46 AM, Hanweidong <hanweidong@xxxxxxxxxx> wrote:
>
>
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxx [mailto:xen-devel-
>> bounces@xxxxxxxxxxxxx] On Behalf Of Hanweidong
>> Sent: 2013å3æ26æ 17:38
>> To: Stefano Stabellini
>> Cc: George Dunlap; xudong.hao@xxxxxxxxx; Yanqiangjun; Luonengjun;
>> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
>> devel@xxxxxxxxxxxxx; xiantao.zhang@xxxxxxxxx
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>>
>> > -----Original Message-----
>> > From: Stefano Stabellini [mailto:stefano.stabellini@xxxxxxxxxxxxx]
>> > Sent: 2013å3æ18æ 20:02
>> > To: Hanweidong
>> > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
>> > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
>> > devel@xxxxxxxxxxxxx; xudong.hao@xxxxxxxxx; xiantao.zhang@xxxxxxxxx
>> > Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
>> > with 4G memory
>> >
>> > On Wed, 13 Mar 2013, Hanweidong wrote:
>> > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
>> > code to init
>> > > RAM in xen_ram_init:
>> > >
>> > >     ...
>> > >     block_len = ram_size;
>> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>> > >         /* Xen does not allocate the memory continuously, and keep
>> a
>> > hole at
>> > >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
>> > >          */
>> > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
>> > >     }
>> > >     memory_region_init_ram(&ram_memory, "xen.ram", block_len);
>> > >     vmstate_register_ram_global(&ram_memory);
>> > >
>> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>> > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
>> > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
>> > >     } else {
>> > >         below_4g_mem_size = ram_size;
>> > >     }
>> > >     ...
>> > >
>> > > HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
>> > to e0000000,
>> > > Which it's consistent with hvmloader when assigning a GPU, and then
>> > guest worked
>> > > for us. So we wondering that xen_ram_init in QEMU should be
>> > consistent with
>> > > hvmloader.
>> > >
>> > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
>> as
>> > below.
>> > > Should keep these places handle the consistent mmio hole or not?
>> > >
>> > >     if (ram_size >= 0xe0000000 ) {
>> > >         above_4g_mem_size = ram_size - 0xe0000000;
>> > >         below_4g_mem_size = 0xe0000000;
>> > >     } else {
>> > >         above_4g_mem_size = 0;
>> > >         below_4g_mem_size = ram_size;
>> > >     }
>> >
>> > The guys at Intel sent a couple of patches recently to fix this issue:
>> >
>> > http://marc.info/?l=xen-devel&m=136150317011027
>> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>> >
>> > Do they solve your problem?
>>
>> These two patches didn't solve our problem.
>>
>
> I debugged this issue with above two patches. I want to share some 
> information and discuss solution here. This issue is actually caused by that 
> a VM has a large pci hole (mmio size) which results in QEMU sets memory 
> regions inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in 
> pc_init1 and xen_ram_init). I created a virtual device with 1GB mmio size to 
> debug this issue. Firstly, QEMU set memory regions except pci hole region in 
> pc_init1() and xen_ram_init(), then hvmloader calculated pci_mem_start as 
> 0x80000000, and wrote it to TOM register, which triggered QEMU to update pci 
> hole region with 0x80000000 using i440fx_update_pci_mem_hole(). Finally the 
> windows 7 VM (configured 8G) crashed with BSOD code 0x00000024. If I hardcode 
> in QEMU pc_init1 and xen_ram_init to match hvmloader's. Then the problem was 
> gone.
>
> Althrough above two patches will pass actual pci hole start address to QEMU, 
> but it's too late, QEMU pc_init1() and xen_ram_init() already set the other 
> memory regions, and obviously the pci hole might overlap with ram regions in 
> this case. So I think hvmloader should setup pci devices and calculate pci 
> hole first, then QEMU can map memory regions correctly from the beginning.

So just to check to make sure I understand correctly: the problem is
that qemu has mapped the guests' pages before the memory hole is made
larger?

In that case, wouldn't it make sense to flush qemu's mapcache whenever
the memory layout changes?  Such a thing should be possible now, e.g.,
when doing ballooning, right?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel