[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] PVHv2/HVM memory layout and interface specification


  • To: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: "Johnson, Ethan" <ejohns48@xxxxxxxxxxxxxxxx>
  • Date: Fri, 26 Jul 2019 19:23:23 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=cs.rochester.edu;dmarc=pass action=none header.from=cs.rochester.edu;dkim=pass header.d=cs.rochester.edu;arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l9yVTtExN7z/z6+7rClQLUGkSXqSBZrnfvbP16lRBnI=; b=PrtZscbdVKbTqk5ykfFJ0RIDj8adRObjdIFvgqT0zsSJPzPxH7x+V2ilw6iWIf3KKq1wzrB1KRfMBhGx9/NODVtfFVe6DUFLz3cILCfztCNNVohUEmGdyTF/ws+JBoOG8mRDwmBqO2ZmtVnxJCe5+omlqvoyCheszxSnTkhOOHUxj4XeLYaMbOH2KCGGdSSDIq7o5CBaG4m003NqcPs1UnGG9v8wvizIWqK2kHVtwgFZmVnMsWtrOMaFDawGXjBJ8Ole/4xLw880WLmkGmtKDNLTq9y+TdbEC6NaQrkiwUt8I3yIkvlBSxpStRboCb3g5l/q4I8YLcRjEu8l4Ax8IQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PfruF2f3phWvRsp3YBrBSygrzbSN0lSXCBeOcsSHvlCN3ltUsKTYK/EYBgPEXvBPacsEMgWwLODkueSjudlJarH1BXKCc/1PfPct8hsH7XKIDfenRYvhEWIEb60VkZHiIErrczAKs1DNTPLbKDElSjsHBxUxsBfRCzkrNKrgKQYsGzAr53lS+Xs4Qo39QzQ6jekJdkGWkUJ+3fVGcskAgv7LNS6woYqQXwctHyeyEM504lMfoEeN2BIcTB3rblDM37mWiXEG2M6VcN3+kNOAv41wgVfjd3zM3+0BUiXKNtd6vdxwdlOd5YiIapaOuIQzB39UnRq+cHDlN47W+LMf1g==
  • Authentication-results: spf=none (sender IP is ) smtp.mailfrom=ejohns48@xxxxxxxxxxxxxxxx;
  • Delivery-date: Fri, 26 Jul 2019 19:23:58 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AdVA3wyWSHObj/sRQ/qradWRmZafIQDCE6+w
  • Thread-topic: PVHv2/HVM memory layout and interface specification

On 2019-07-22 21:11:40 -0400, Andrew Cooper wrote:
> On 23/07/2019 00:50, Johnson, Ethan wrote:
> > (The Xen 4.7 version of the documentation had a little bit
> > more detail at https://xenbits.xen.org/docs/4.7-testing/misc/pvh.html, but
> > it still doesn't have quite what I'm looking for, and I'm not sure if
> > that's for PVHv2 or the older PVHv1.)
> 
> That document was for PVHv1 which is completely obsolete and gone now. 
> As for these days...
> 
> Xen has two fundamental types of virtualisation.  Ring deprivileging
> which is what PV guests use, and hardware extensions (Intel VT-x, AMD
> SVM) which is what HVM guests use.  PVH is identical to HVM in this
> regard; what distinguishes PVH is that there is no Qemu providing an
> emulation of a IBM-compatible PC behind the scenes.
> 
> PV guests share an address space with Xen.  In this regard, they are
> very similar to userspace sharing pagetables with the guest kernel.  In
> our case, we distinguish guest user, guest kernel, and Xen, but the same
> basic principles hold.
> 
> HVM guests get an entirely separate address space, either provided by
> hardware extensions (Intel EPT, AMD NPT), or via shadow pagetables. 
> Either way, Xen itself is not a part of this address space, but does
> manage it from the outside.
> 
> > 1. Where does the Xen hypervisor itself live in the host's (VMX root)
> > virtual and physical address spaces under PVH and HVM? In classic PV it
> > seems it lived just below the guest kernel, but that was without extended
> > paging separating the host and guest's views of memory.
> 
> Where exactly Xen lives is VMX root mode is of no concern to an HVM/PVH
> guest.  That said, it is in practice the same as for PV so we can run PV
> and HVM/PVH guests at the same time.  (Its only recently we've introduce
> the ability to selectively compile out PV or HVM support.)
> 
> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/asm-x86/config.h;h=9ef9d03ca7199ccca416c6ea2d24adaf6dbc4e0f;hb=refs/heads/staging#l116
> describes Xen's virtual memory layout, including the bits applicable to
> PV guests.
> 
> Virtually, Xen lives in the first 8Tb of address space of the upper
> canonical half, while physically, the only restriction is that it must
> be in the bottom 4G of RAM.  The exact location is machine specific, but
> defaults to highest (sufficiently sized) block of RAM below the 4G boundary.
> 
> > 2. Does Xen also live somewhere within the guest's (VMX non-root) view of
> > physical memory as provided by extended paging, in PVH mode? (I'm guessing
> > it wouldn't for HVM, but I'm wondering about PVH because it did for classic
> > PV.)
> 
> No.  For PVH (technically v2, but we don't call it that any more), think
> "HVM without Qemu".
> 
> PVHv1 was very misguided, which is why it no longer exists.
> 
> > 3. How are hypercalls handled in PVH/HVM mode? Do they use the VMCALL 
> > instruction (or something comparable) to initiate a VM exit, or are they 
> > sometimes handled within VMX non-root mode without a VM exit, similar to 
> > classic PV?
> 
> Ah - this document is one I prepared earlier.  To answer your lower
> question, our docs are currently terrible, and I'm trying to improve
> things.  This is the first part of what I hope will be a document that
> would eventually cover all of your questions.  Feedback welcome,
> including if there is anything unclear in the few bits which currently
> exist.
> 
> http://xenbits.xen.org/docs/sphinx-unstable/guest-guide/x86/hypercall-abi.html
> 
> 
> > 4. How does Xen access the guest's memory when it needs to examine or
> > modify it? Does it just access the physical frames through a direct map in
> > VMX root mode, or are guests' views of physical memory somehow represented
> > somewhere in the host's (hypervisor's) virtual address space?
> 
> Xen has a directmap of host memory, but we're looking to reduce the use
> of that as a defence against speculative sidechannels.  Xen does not
> have a 1:1 directmap of guest physical address space (i.e. we don't have
> anything like Qemu/KVM has).
> 
> All physical accesses into guest memory start by walking the P2M (the
> EPT/NPT or Shadow pagetables) to find the target data.  For accesses
> which are based on virtual addresses rather than guest physical, we
> perform a pagewalk of the guests pagetables to arrive at the target.
> 
> For emulated MMIO, frames may not be present, at which case special
> handling kicks in, such as forwarding the IO request to Qemu, or
> terminating it with default x86 behaviour (read ~0, write discard).
> 
> > 5. How do PVH guests learn what their physical memory layout looks like? 
> > Which regions of memory do they use to communicate with Xen? How is this 
> > process different for HVM (non-PVH) guests?
> 
> By default, the E820 table passed at boot in the PVH start_info
> structure.  There is also a hypercall which can be used to obtain a copy
> of the E820 table which the toolstack produced.
> 
> > 6. How does dom0 factor into all this? Does it have any direct access to 
> > domUs' physical memory pages, or does it have to ask Xen to do that on its 
> > behalf with a hypercall?
> 
> Dom0 is just a plain VM, with some default permissions.  In this regard,
> it is just like a root process running with more privilege than regular
> userspace.
> 
> One concept Xen has is that of a "foreign memory map" which allows dom0
> (or other sufficiently privileged domains) to map guest memory.  This is
> used for all kinds of tasks, from booting the guest in the first place,
> to live migration, and for Qemu to emulate DMA access for an HVM guest.
> 
> > Thanks in advance for any help anyone can offer. If there are any good 
> > resources out there (books? blog posts?) for understanding the big-picture 
> > structure of the Xen architecture and codebase as it exists in its modern 
> > form (i.e., in the PVH/HVM era), those would be especially appreciated.
> 
> I am trying to make things better.  For now, asking here is probably
> your best bet.
> 
> In some copious free time, I'll try translating this into the guest
> guide which was linked above.
> 
> I hope this is all clear and easy to follow.
> 
> ~Andrew

Thanks for taking the time to write out all these great answers - this is
exactly what I was looking for!

(Apologies for not responding sooner; for whatever reason, your reply didn't
make it to my inbox and I had to find it on the list's web archives. I
suspect one of my own inbox filters I set up to winnow this high-traffic list is
to blame...)

Sincerely,
Ethan Johnson

-- 
Ethan J. Johnson
Computer Science PhD student, Systems group, University of Rochester
ejohns48@xxxxxxxxxxxxxxxx
ethanjohnson@xxxxxxx
PGP public key available from public directory or on request

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.