Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space

[Oleksandr <olekstysh@xxxxxxxxx>]

On 09.08.21 23:45, Julien Grall wrote:


Hi Julien

> On 09/08/2021 19:24, Oleksandr wrote:
> > On 09.08.21 18:42, Julien Grall wrote:
> > > Hi Oleksandr,
> >
> > Hi Julien.
> >
> >
> > Thank you for the input.
> >
> >
> > > On 07/08/2021 18:03, Oleksandr wrote:
> > > > On 06.08.21 03:30, Stefano Stabellini wrote:
> > > >
> > > > Hi Stefano
> > > >
> > > > > On Wed, 4 Aug 2021, Julien Grall wrote:
> > > > > > > +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 
> > > > > > > 128GB */
> > > > > > > +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
> > > > > > >
> > > > > > > While the possible new DT bindings has not been agreed yet, I 
> > > > > > > re-used
> > > > > > > existing "reg" property under the hypervisor node to pass safe 
> > > > > > > range as a
> > > > > > > second region,
> > > > > > > https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: 
> > > > > > So a single region works for a guest today, but for dom0 we will 
> > > > > > need multiple
> > > > > > regions because it is may be difficult to find enough contiguous 
> > > > > > space for a
> > > > > > single region.
> > > > > >
> > > > > > That said, as dom0 is mapped 1:1 (including some guest mapping), 
> > > > > > there is also
> > > > > > the question where to allocate the safe region. For grant table, 
> > > > > > we so far
> > > > > > re-use the Xen address space because it is assumed it will space 
> > > > > > will always
> > > > > > be bigger than the grant table.
> > > > > >
> > > > > > I am not sure yet where we could allocate the safe regions. 
> > > > > > Stefano, do you
> > > > > > have any ideas?
> > > > > The safest choice would be the address range corresponding to memory
> > > > > (/memory) not already allocated to Dom0.
> > > > >
> > > > > For instance from my last boot logs:
> > > > > (XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
> > > > > (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB)
> > > > > (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
> > > > >
> > > > > All the other ranges could be given as unallocated space:
> > > > >
> > > > > - 0x0 - 0x10000000
> > > > > - 0x70000000 - 0x78000000
> > > > > - 0x8_0000_0000 - 0x8_8000_0000
> > > > Thank you for the ideas.
> > > >
> > > > If I got the idea correctly, yes, as these ranges represent the 
> > > > real RAM, so no I/O would be in conflict with them and as the 
> > > > result - no overlaps would be expected.
> > > > But, I wonder, would this work if we have IOMMU enabled for Dom0 
> > > > and need to establish 1:1 mapping for the DMA devices to work with 
> > > > grant mappings...
> > > > In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn 
> > > > = mfn, so the question is could we end up with this new gfn 
> > > > replacing the valid mapping
> > > > (with gfn allocated from the safe region)?
> > > Right, when we enable the IOMMU for dom0, Xen will add an extra 
> > > mapping with GFN == MFN for foreign and grant pages. This is because 
> > > Linux is not aware that whether a device is protected by an IOMMU. 
> > > Therefore it is assuming it is not and will use the MFN to configure 
> > > for DMA transaction.
> > > We can't remove the mapping without significant changes in Linux and 
> > > Xen. I would not mandate them for this work.
> > > That said, I think it would be acceptable to have different way to 
> > > find the region depending on the dom0 configuration. So we could use 
> > > the RAM not used by dom0 when the IOMMU is turned off.
> > OK
> >
> >
> > > > > The second best choice would be an hole: an address range not used by
> > > > > anybody else (no reg property) and also not even mappable by a bus 
> > > > > (not
> > > > > covered by a ranges property). This is not the best choice because 
> > > > > there
> > > > > can cases where physical resources appear afterwards.
> > > Are you saying that the original device-tree doesn't even describe 
> > > them in any way (i.e. reserved...)?
> > > > Unfortunately, yes.
> > > So the decision where the safe region is located will be done by 
> > > Xen. There is no involvement of the domain (it will discover the 
> > > region from the DT). Therefore, I don't think we need to think about 
> > > everything right now as we could adapt this is exact region is not 
> > > part of the stable ABI.
> > > The hotplug is one I would defer because this is not supported (and 
> > > quite likely not working) in Xen upstream today.
> > Sounds reasonable.
> >
> >
> > > Now regarding the case where dom0 is using the IOMMU. The assumption 
> > > is Xen will be able to figure out all the regions used from the 
> > > firmware table (ACPI or DT).
> > > AFAIK, this assumption would be correct for DT. However, for ACPI, I 
> > > remember we were not able to find all the MMIOs region in Xen (see 
> > > [1] and [2]). So even this solution would not work for ACPI.
> > > If I am not mistaken, we don't support IOMMU with ACPI yet. So we 
> > > could defer the problem to when this is going to be supported.
> > Sounds reasonable.
> >
> >
> > To summarize:
> >
> > 0. Skip ACPI case for now, implement for DT case
> Just to be clear, I suggested to skip it when the IOMMU is enabled 
> with ACPI. We should still support the case without IOMMU. The 
> implementation would be the same as 2.
yes, sorry for not being precise


> > 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as 
> > safe ranges
> > I would take into the account holes >= 1MB.
> May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which 
would not be too useful overall, but could bloat resulting reg property.
Anyway, I would be ok with any sizes.


> > I am wondering, do we need a special alignment here other than a 
> > PAGE_SIZE?
> It needs to be 64KB aligned so a guest using 64KB pages can use it.
ok, sounds reasonable


> > 2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to 
> > Dom0 as safe ranges
> > We could even provide holes here as well.
> I would rather not. We likely need hack for the hotplug case. So I 
> want to keep them contained to IOMMU unless there is a strong reason 
> to do it.
ok, I got it


> Cheers,
--
Regards,

Oleksandr Tyshchenko