Re: [Xen-devel] (v2) Design proposal for RMRR fix

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

> From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
> Sent: Tuesday, January 13, 2015 11:59 PM
> 
> On 01/13/2015 03:52 PM, Jan Beulich wrote:
> >>>> On 13.01.15 at 13:03, <kevin.tian@xxxxxxxxx> wrote:
> >>>  From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
> >>> Sent: Tuesday, January 13, 2015 7:56 PM
> >>>>>> On 13.01.15 at 12:03, <kevin.tian@xxxxxxxxx> wrote:
> >>>>  lowmem:         [0, 0x9fffffff]
> >>>>  mmio hole:      [0xa0000000, 0xffffffff]
> >>>>  highmem:        [0x100000000, 0x160000000]
> >>>>
> >>>>
> >>>> For [0x40000000, 0x40003fff], leave it as a conflict since either
> >>>> reducing lowmem to 1G is not nice to guest which doesn't use
> >>>> highmem or we have to break lowmem into two trunks so more
> >>>> structure changes are required.
> >>>
> >>> This makes no sense - if such an area was explicitly requested to
> >>> be reserved, leaving it as a conflict is not an option.
> >>
> >> explicitly requested by libxl but leaving it as a conflict in domain
> >> builder is just fine. later steps will actually catch conflicts when
> >> relevant regions are actually used (e.g. in static assignment, in
> >> hotplug, or in migration).
> >
> > But why do you think xl requested the region to be reserved?
> > Presumably because the guest config file said so. And if the
> > config file said so, there's no alternative to punching a hole
> > there - failing device assignment later on is what the guest
> > config file setting was added to avoid.
> 
> Yes -- the general principle should be: if the admin has asked for X,
> and X cannot be done, then the entire operation should fail, so that the
> admin can either make it possible for X to happen, or decide to do Y
> instead.  Automatically doing Z when the admin has explicitly asked for
> X is poor interface design.
> 
> That's why libxl will destroy the domain if the pci_add fails: if you've
> asked for device A to be assigned (X), and it can't be assigned for
> whatever reason, the domain creation should fail and the admin should be
> told why, so he can ask for what he wants instead.
> 
> In the case of RMRRs, if the admin has asked for an RMRR to be made, and
> it can't be made for whatever reason, the guest creation should fail.
> For statically assigned devices this will fall out naturally from the
> device assignment failing; but for RMRRs corresponding to hot-plug
> regions, we'll have to fail somewhere else -- preferrably in libxc
> rather than hvmloader?
> 
> We may want to make a special case for RMRRs that overlap guest BIOS I
> guess.

Apparently I didn't get support on a simplified model which only warns conflict
in domain builder while letting later assignment to actual fail. As you pointed 
here, it violates the general principle. That's fine.

So I'll withdraw my proposal and then discuss detail along your idea.

Now assume high level libxl will query all host RMRRs from Xen hypervisor,
and provide one option for rmrr_host (requiring all host RMRRs for hotplug 
preparation) and also another option to allow specify additional regions 
which may come from other host due to migration (for static-assigned 
device we don't need a new option). high level libxl will decide a set of
reserved regions according to user configurations or whatever policies, and
pass those information to domain builder.

So domain builder doesn't know the association between devices and specified 
regions (in migration there won't be such association). It just tries to detect 
and make holes to avoid potential conflicts, and if can't then fail the whole 
domain creation at all per your suggestions.

We discussed earlier there are two reasons that some conflicts may not be 
avoided:
        - RMRRs conflicting with guest BIOS in <1MB area, as an example of 
hard conflicts
        - RMRRs conflicting with lowmem which is low enough then avoiding it
will either break lowmem or make lowmem too low to impact guest (just
an option being discussed)

Now the open is whether we want to fail domain creation for all of above
conflicts. user may choose to bear with conflicts at his own disposal, or
libxl doesn't want to fail conflicts as preparation for future 
hotplug/migration.
One possible option is to add a per-region flag to specify whether treating
relevant conflict as an error, when libxl composes the list to domain builder. 
and this information will be saved in a user space database accessible to
all components and also waterfall to Xen hypervisor when libxl requests 
actual device assignment.

with such model to tell whether conflict is treated as an error for a 
specific region, we can get unified policy cross different components
(libxc/Xen/hvmloader).

for another open of having the domain builder do minimal work to launch
hvmloader who does actual population, it's a good idea but I'm not sure
how much work remains in the direction. It's a bigger scope change than
what's anticipated for this RMRR work and the detail hasn't been discussed
or agreed by toolstack experts. So I'd propose sticking to original proposal 
to make this RMRR fix a reasonably necessary task (i.e. let domain builder 
handle RAM conflicts while let hvmloader handle other conflicts), instead 
of growing it unboundly into a more general cleanup. If that's the right
direction to go, we also hope other experts can jump in to help. :-)

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel