Re: [Xen-devel] (v2) Design proposal for RMRR fix

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Mon, Jan 12, 2015 at 11:25:56AM +0000, George Dunlap wrote:
> On Fri, Jan 9, 2015 at 2:43 AM, Tian, Kevin <kevin.tian@xxxxxxxxx> wrote:
> >> From: George Dunlap
> >> Sent: Thursday, January 08, 2015 8:55 PM
> >>
> >> On Thu, Jan 8, 2015 at 12:49 PM, George Dunlap
> >> <George.Dunlap@xxxxxxxxxxxxx> wrote:
> >> > If RMRRs almost always happen up above 2G, for example, then a simple
> >> > solution that wouldn't require too much work would be to make sure
> >> > that the PCI MMIO hole we specify to libxc and to qemu-upstream is big
> >> > enough to include all RMRRs.  That would satisfy the libxc and qemu
> >> > requirements.
> >> >
> >> > If we then store specific RMRRs we want included in xenstore,
> >> > hvmloader can put them in the e820 map, and that would satisfy the
> >> > hvmloader requirement.
> >>
> >> An alternate thing to do here would be to "properly" fix the
> >> qemu-upstream problem, by making a way for hvmloader to communicate
> >> changes in the gpfn layout to qemu.
> >>
> >> Then hvmloader could do the work of moving memory under RMRRs to
> >> higher memory; and libxc wouldn't need to be involved at all.
> >>
> >> I think it would also fix our long-standing issues with assigning PCI
> >> devices to qemu-upstream guests, which up until now have only been
> >> worked around.
> >>
> >
> > could you elaborate a bit for that long-standing issue?
> 
> So qemu-traditional didn't particularly expect to know the guest
> memory layout.  qemu-upstream does; it expects to know what areas of
> memory are guest memory and what areas of memory are unmapped.  If a
> read or write happens to a gpfn which *xen* knows is valid, but which
> *qemu-upstream* thinks is unmapped, then qemu-upstream will crash.
> 
> The problem though is that the guest's memory map is not actually
> communicated to qemu-upstream in any way.  Originally, qemu-upstream
> was only told how much memory the guest had, and it just "happens" to
> choose the same guest memory layout as the libxc domain builder does.
> This works, but it is bad design, because if libxc were to change for
> some reason, people would have to simply remember to also change the
> qemu-upstream layout.
> 
> Where this really bites us is in PCI pass-through.  The default <4G
> MMIO hole is very small; and hvmloader naturally expects to be able to
> make this area larger by relocating memory from below 4G to above 4G.
> It moves the memory in Xen's p2m, but it has no way of communicating
> this to qemu-upstream.  So when the guest does an MMIO instuction that
> causes qemu-upstream to access that memory, the guest crashes.
> 
> There are two work-arounds at the moment:
> 1. A flag which tells hvmloader not to relocate memory
> 2. The option to tell qemu-upstream to make the memory hole larger.
> 
> Both are just work-arounds though; a "proper fix" would be to allow
> hvmloader some way of telling qemu that the memory has moved, so it
> can update its memory map.
> 
> This will (I'm pretty sure) have an effect on RMRR regions as well,
> for the reasons I've mentioned above: whether make the "holes" for the
> RMRRs in libxc or in hvmloader, if we *move* that memory up to the top
> of the address space (rather than, say, just not giving that RAM to
> the guest), then qemu-upstream's idea of the guest memory map will be
> wrong, and will probably crash at some point.
> 
> Having the ability for hvmloader to populate and/or move the memory
> around, and then tell qemu-upstream what the resulting map looked
> like, would fix both the MMIO-resize issue and the RMRR problem, wrt
> qemu-upstream.
> 

Hmm, wasn't this changed slightly during Xen 4.5 development by Don Slutz?

You can now specify the mmio_hole size for HVM guests when using qemu-upstream:
http://wiki.xenproject.org/wiki/Xen_Project_4.5_Feature_List


"Bigger PCI MMIO hole in QEMU via the mmio_hole parameter in guest config, 
which allows configuring the MMIO size below 4GB. "

"Backport pc & q35: Add new machine opt max-ram-below-4g":
http://xenbits.xen.org/gitweb/?p=qemu-upstream-unstable.git;a=commit;h=ffdacad07002e14a8072ae28086a57452e48d458

"x86: hvm: Allow configuration of the size of the mmio_hole.":
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=2d927fc41b8e130b3b8910e4442d4691111d2ac7


-- Pasi


>  -George
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel