Re: [Xen-devel] Question about xenpage_list

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 28/08/2019 18:35, Tamas K Lengyel wrote:
> On Wed, Aug 28, 2019 at 11:16 AM Andrew Cooper
> <andrew.cooper3@xxxxxxxxxx> wrote:
>> On 28/08/2019 18:07, Tamas K Lengyel wrote:
>>> On Wed, Aug 28, 2019 at 10:55 AM Andrew Cooper
>>> <andrew.cooper3@xxxxxxxxxx> wrote:
>>>> On 28/08/2019 17:25, Tamas K Lengyel wrote:
>>>>> On Wed, Aug 28, 2019 at 9:54 AM Jan Beulich <jbeulich@xxxxxxxx> wrote:
>>>>>> On 28.08.2019 17:51, Tamas K Lengyel wrote:
>>>>>>> On Wed, Aug 28, 2019 at 9:35 AM Jan Beulich <jbeulich@xxxxxxxx> wrote:
>>>>>>>> On 28.08.2019 17:28, Tamas K Lengyel wrote:
>>>>>>>>> Hi all,
>>>>>>>>> I'm trying to track down how a call in common/grant_table.c to
>>>>>>>>> share_xen_page_with_guest will actually populate that page into the
>>>>>>>>> guest's physmap.
>>>> share_xen_page_with_guest() is perhaps poorly named.  It makes the page
>>>> able to be inserted into the guests p2m.
>>>>
>>>> It is internal accounting, so that the permission checks in a subsequent
>>>> add_to_physmap() call will pass.
>>>>
>>>> Perhaps it should be named "allow_guest_access_to_frame()" or similar.
>>>>
>>>>>>>>>  Immediately after the call the page doesn't seem to
>>>>>>>>> be present in the physmap, as share_xen_page_with_guest will just add
>>>>>>>>> the page to the domain's xenpage_list linked-list:
>>>>>>>>>
>>>>>>>>>         unsigned long mfn;
>>>>>>>>>         unsigned long gfn;
>>>>>>>>>
>>>>>>>>>         share_xen_page_with_guest(virt_to_page(gt->shared_raw[i]), d, 
>>>>>>>>> SHARE_rw);
>>>>>>>>>
>>>>>>>>>         mfn = virt_to_mfn(gt->shared_raw[i]);
>>>>>>>>>         gfn = mfn_to_gmfn(d, mfn);
>>>>>>>>>
>>>>>>>>>         gdprintk(XENLOG_INFO, "Sharing %lx -> %lx with domain %u\n",
>>>>>>>>> gfn, mfn, d->domain_id);
>>>>>>>>>
>>>>>>>>> This results in the following:
>>>>>>>>>
>>>>>>>>> (XEN) grant_table.c:1820:d0v0 Sharing ffffffffffffffff -> 42c71e with 
>>>>>>>>> domain 1
>>>>>>>>>
>>>>>>>>> AFAICT the page only gets populated into the physmap once the domain
>>>>>>>>> gets unpaused. If I let the domain run and then pause it I can see
>>>>>>>>> that the page is in the guest's physmap (by looping through all the
>>>>>>>>> entries in its EPT):
>>>>>>>>>
>>>>>>>>> (XEN) mem_sharing.c:1636:d0v0 0xf2000 -> 0x42c71e is a grant mapping
>>>>>>>>> shared with the guest
>>>>>>>> This should be the result of the domain having made a respective
>>>>>>>> XENMAPSPACE_grant_table request, shouldn't it?
>>>>>>>>
>>>>>>> Do you mean the guest itself or the toolstack?
>>>>>> The guest itself - how would the tool stack know where to put the
>>>>>> frame(s)?
>>>>> I don't think that makes sense. How would a guest itself now that it
>>>>> needs to map that frame? When you restore the VM from a savefile, it
>>>>> is already running and no firmware is going to run in it to initialize
>>>>> such GFNs.
>>>>>
>>>>> As for the toolstack, I see calls to xc_dom_gnttab_seed from the
>>>>> toolstack during domain creation (be it a new domain or one being
>>>>> restored from a save file) which does issue a XENMEM_add_to_physmap
>>>>> with the space being specified as XENMAPSPACE_grant_table. Looks like
>>>>> it gathers the GFN via xc_core_arch_get_scratch_gpfn. So it looks like
>>>>> that's how its done.
>>>> On domain creation, the toolstack needs to write the store/console grant
>>>> entry.
>>>>
>>>> If XENMEM_acquire_resource is available and usable (needs newish Xen and
>>>> dom0 kernel), then that method is preferred.  This lets the toolstack
>>>> map the grant table frame directly, without inserting it into the guests
>>>> p2m first.
>>>>
>>>> The fallback path is to pick a free pfn, insert it into the guest
>>>> physmap, foreign map it, write the entries, unmap and remove from the
>>>> guest physmap.  This has various poor properties like shattering
>>>> superpages for the guest, and a general inability to function correctly
>>>> once the guest has started executing and has a balloon driver in place.
>>>>
>>>> At a later point, once the guest starts executing, a grant-table aware
>>>> part of the kernel ought to map the grant table at the kernels preferred
>>>> location and keep it there permanently.
>>>>
>>> OK, makes sense, but when the guest is being restored from a savefile,
>>> how does it know that it needs to do that mapping again? That frame is
>>> being re-created during restoration, so when the guest starts to
>>> execute again it would just have a whole where that page used to be.
>> This is where we get to the problems of Xen's "migration" not being
>> transparent.  Currently it is the requirement of the guest kernel to
>> remap the grant table on resume.
>>
>> This is a reasonable requirement for PV guests.  Because PV guest
>> kernels maintain their own P2M, it is impossible to migrate transparently.
>>
>> This should never have made it into the HVM ABI, but it did and we're a
>> decade too late, and only just starting to pick up the pieces.
>>
>> I presume you're doing some paging work here, and are logically
>> restoring a guest without its knowledge?
>>
> Correct, I'm creating a VM by populating its physmap with mem_shared
> entries from another domain that's paused by looping through all pages
> and memsharing them. Pages that are not sharable I manually allocate
> new pages for and copy them over (or simply plug the GFN in with
> INVALID_MFN if the type is such that it allows that). Currently this
> works fine when the domain I'm populating from was just restored from
> a savefile, including launching the toolstack for the new domain and
> interacting with its VNC/network/etc. But I'm running into trouble
> when the domain I'm copying from was unpaused before this sharing is
> taking place. Evidently unpausing the domain introduces discrepancies
> in its memory space as compared to when its just restored and this
> grant mapping is one page that pops up as being mapped in but it's
> unsharable since it's a PGC_xen_heap page. The new domain does have
> the page allocated already but it doesn't have a GFN yet, so I can't
> copy its content over since I have no way to ensure that MFN will be
> used for the same GFN. There are other pages that also pop up as now
> not being shared, but seems like running a domain_soft_reset on the
> domain makes them go away (resetting event channels in particular). So
> anyway, I'm having a hard time figuring out what changes are made to
> the domain after it is unpaused, as I need to revert them all to make
> the domain be in the same state it would be when restored from the
> savefile.

Its not safe to blindly copy any xenheap page, irrespective of whether
they are present in the p2m or not.

For example, grants which are mapped in paused domain need to appear as
unmapped in the cloned domain, or Xen is going to object to the the
state it finds the new grant table in.

It sounds as if you want an explicitly "duplicate xenheap pages" step
which isn't a straight forward memcpy.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel