Re: Mapping memory into a domain

[Demi Marie Obenour <demiobenour@xxxxxxxxx>]

On 5/6/25 9:06 AM, Alejandro Vallejo wrote:
> On Tue May 6, 2025 at 3:02 AM CEST, Demi Marie Obenour wrote:
>> On 5/5/25 7:32 AM, Alejandro Vallejo wrote:
>>> I suppose this is still about multiplexing the GPU driver the way we
>>> last discussed at Xen Summit?
>>>
>>> On Mon May 5, 2025 at 12:51 AM CEST, Demi Marie Obenour wrote:
>>>> What are the appropriate Xen internal functions for:
>>>>
>>>> 1. Turning a PFN into an MFN?
>>>> 2. Mapping an MFN into a guest?
>>>> 3. Unmapping that MFN from a guest?
>>>
>>> The p2m is the single source of truth about such mappings.
>>>
>>> This is all racy business. You want to keep the p2m lock for the full
>>> duration of whatever operation you wish do, or you risk another CPU
>>> taking it and pulling the rug under your feet at the most inconvenient
>>> time.
>>>
>>> In general all this faff is hidden under way too many layers beneath
>>> copy_{to,from}_guest(). Other p2m manipulation high-level constructs
>>> that might do interesting things worth looking at may be 
>>> {map,unmap}_mmio_region()
>>>
>>> Note that not every pfn has an associated mfn. Not even every valid pfn
>>> has necessarily an associated mfn (there's pod). And all of this is
>>> volatile business in the presence of a baloon driver or vPCI placing
>>> mmio windows over guest memory.
>>
>> Can I check that POD is not in use?  
> 
> Maybe, but now you're reaching exponential complexity considering each
> individual knob of the p2m into account.
> 
>>
>>> In general anything up this alley would need a cohesive pair for
>>> map/unmap and a credible plan for concurrency and how it's all handled
>>> in conjunction with other bits that touch the p2m.
>>
>> Is taking the p2m lock for the entire operation a reasonable approach
>> for concurrency?  Will this cause too much lock contention?
> 
> Maybe. It'd be fine for a page. Likely not so for several GiB if they
> aren't already superpages.
> 
>>
>>>> The first patch I am going to send with this information is a documentation
>>>> patch so that others do not need to figure this out for themselves.
>>>> I remember being unsure even after looking through the source code, which
>>>> is why I am asking here.
>>>
>>> That's not surprising. There's per-arch stuff, per-p2mtype stuff,
>>> per-guesttype stuff. Plus madness like on-demand memory. It's no wonder
>>> such helpers don't exist and the general manipulations are hard to
>>> explain.
>>
>> Is this a task that is only suitable for someone who has several years
>> experience working on Xen, or is it something that would make sense for
>> someone who is less experienced?
> 
> The p2m is a very complex beast that integrates more features than I
> care to count. It requires a lot of prior knowledge. Whoever does it
> must know Xen fairly well in many configurations.
> 
> The real problem is finding the right primitives that do what you want
> without overcomplicating everything else, preserving system security
> invariants and have benign (and ideally clear) edge cases.
> 
> This was the last email you sent (I think?). Has any of the requirements
> changed in any direction?
> 
>   https://lore.kernel.org/xen-devel/Z5794ysNE4KDkFuT@itl-email/

Map and Revoke are still needed, with the same requirements as described
in this email.  Steal and Return were needed for GPU shared virtual memory,
but it has been decided to not support this with virtio-GPU, so these
primitives are no longer needed.

> Something I'm missing there is how everything works without Xen. That
> might help (me, at least) guage what could prove enough to support the
> usecase. Are there sequence diagrams anywhere about how this whole thing
> works without Xen? I vaguely remember you showing something last year in
> Xen Summit in the design session, but my memory isn't that good :)

A Linux driver that needs access to userspace memory
pages can get it in two different ways:

1. It can pin the pages using the pin_user_pages family of APIs.
   If these functions succeed, the driver is guaranteed to be able
   to access the pages until it unpins them.  However, this also
   means that the pages cannot be paged out or migrated.  Furthermore,
   file-backed pages cannot be safely pinned, and pinning GPU memory
   isn’t supported.  (At a minimum, it would prevent the pages from
   migrating from system RAM to VRAM, so all access by a dGPU would
   cross the PCIe bus, which would be very slow.)

2. It can grab the *current* location of the pages and register an
   MMU notifier.  This works for GPU memory and file-backed memory.
   However, when the invalidate_range function of this callback, the
   driver *must* stop all further accesses to the pages.

   The invalidate_range callback is not allowed to block for a long
   period of time.  My understanding is that things like dirty page
   writeback are blocked while the callback is in progress.  My
   understanding is also that the callback is not allowed to fail.
   I believe it can return a retryable error but I don’t think that
   it is allowed to keep failing forever.

   Linux’s grant table driver actually had a bug in this area, which
   led to deadlocks.  I fixed that a while back.

KVM implements the second option: it maps pages into the stage-2
page tables (or shadow page tables, if that is chosen) and unmaps
them when the invalidate_range callback is called.  Furthermore,
if a page fault happens while the page is unmapped, KVM will try
to bring the pages back into memory so the guest can access it.

For GPU acceleration via virtio-GPU native contexts to work,
the Xen interface driver needs to do the same thing with GPU
buffers that KVM does: it needs to fault the pages into guest
memory on-demand and revoke access to the pages when the host
kernel demands them back.  There really is no alternative that
I am aware of.  The need to handle guest page faults doesn’t
come from the host kernel, but rather from guest userspace.
It isn’t practical to change guest userspace to remove this
requirement.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

Attachment: OpenPGP_0xB288B55FFF9C22C1.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature