Re: [Xen-devel] PCIe devices that are hotplugged after MMIO has been setup fail due to _CRS not covering 64-bit area

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 12.10.16 at 23:15, <konrad.wilk@xxxxxxxxxx> wrote:
> On Wed, Sep 28, 2016 at 03:21:08AM -0600, Jan Beulich wrote:
>> >>> On 27.09.16 at 16:43, <konrad.wilk@xxxxxxxxxx> wrote:
>> > If the guest is booted with 'pci' we nicely expand the MMIO region below
>> > 4GB and try to fit in the BARs in there. If that fails (not enough
>> > space) we move it above the memory (64-bit). And throughout all of this
>> > we also update the _CRS field to cover these ranges.
>> > 
>> > (Note, I need to check if the 64-bit area is also set, I think it is).
>> > 
>> > But the situation is different if we hot-plug a device that has too big
>> > BAR to fit in the MMIO region. We move it in the 64-bit area but we
>> > don't update the _CRS. Which means that Linux will complain (unless
>> > booted with pci=nocrs)). Not sure about Windows but I would assume so
>> > to.
>> > 
>> > I was wondering what would be a good way to solve this? I looked at some
>> > Dell machines to see how they deal with hotplug PCIe devices and they
>> > just declared all the memory in the _CRS (including RAM).
>> > 
>> > We could do a hybrid - during bootup make the _CRS region have entry from
>> > end of RAM to .. end of memory?
>> 
>> End of physical address space you mean? Generally yes, but we
>> need to be a little careful there: For one, on AMD we'd better not
>> overlap with the HT area. And then there's this MTRR related
>> comment next to the setting of pci_hi_mem_end (albeit both HT
>> area start and end of PA space should be aligned well enough).
>> 
>> > Or perhaps add some extra logic between QEMU and ACPI AML to expand (or
>> > perhaps modify the last _CRS entry) when PCIe devices are hotplugged?
>> 
>> While that would be the most flexible variant, I'd be afraid of this
>> getting rather complicated. Or have you already got some
>> reasonable layout of how this would look like?
> 
> I did this and while all the plumbing works great and I can see that
> the pci_hi_len gets incremented by the size of the 64-bit BARS of the
> new device (and also decremented if hot-unplugged) I hit a snag:
> 
> Linux evaluates this only once (actually twice, but only during bootup).

Ah - quite reasonable to expect this won't change.

> For right now let me jump with the "simpler" solution of just
> hardcoding the end of physical address space and see how that works out.

Right.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel