Re: [openxt-dev] VirtIO-Argo initial development proposal

[Christopher Clark <christopher.w.clark@xxxxxxxxx>]

On Thu, Dec 31, 2020 at 3:39 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Thu, Dec 31, 2020 at 11:02:40AM +0000, Julien Grall wrote:
> > On Thu, 31 Dec 2020 at 08:46, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> > >
> > > On Wed, Dec 30, 2020 at 11:30:03AM +0000, Julien Grall wrote:
> > > > Hi Roger,
> > > >
> > > > On 29/12/2020 09:17, Roger Pau Monné wrote:
> > > > > On Wed, Dec 23, 2020 at 04:32:01PM -0500, Rich Persaud wrote:
> > > > > > On Dec 17, 2020, at 07:13, Jean-Philippe Ouellet <jpo@xxxxxx> 
> > > > > > wrote:
> > > > > > > On Wed, Dec 16, 2020 at 2:37 PM Christopher Clark
> > > > > > > <christopher.w.clark@xxxxxxxxx> wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have written a page for the OpenXT wiki describing a proposal 
> > > > > > > > for
> > > > > > > > initial development towards the VirtIO-Argo transport driver, 
> > > > > > > > and the
> > > > > > > > related system components to support it, destined for OpenXT and
> > > > > > > > upstream projects:
> > > > > > > >
> > > > > > > > https://openxt.atlassian.net/wiki/spaces/~cclark/pages/1696169985/VirtIO-Argo+Development+Phase+1
> > > > >
> > > > > Thanks for the detailed document, I've taken a look and there's indeed
> > > > > a lot of work to do listed there :). I have some suggestion and
> > > > > questions.
> > > > >
> > > > > Overall I think it would be easier for VirtIO to take a new transport
> > > > > if it's not tied to a specific hypervisor. The way Argo is implemented
> > > > > right now is using hypercalls, which is a mechanism specific to Xen.
> > > > The concept of hypervisor call is not Xen specific. Any other 
> > > > hypervisor can
> > > > easily implement them. At least this is the case on Arm because we have 
> > > > an
> > > > instruction 'hvc' that acts the same way as a syscall but for the
> > > > hypervisor.
> > > >
> > > > What we would need to do is reserving a range for Argos. It should be
> > > > possible to do it on Arm thanks to the SMCCC (see [1]).
> > > >
> > > > I am not sure whether you have something similar on x86.
> > >
> > > On x86 Intel has vmcall and AMD vmmcall, but those are only available
> > > to HVM guests.
> >
> > Right, as it would for other architecture if one decided to implement
> > PV (or binary translated) guests.
> > While it may be possible to use a different way to communicate on x86
> > (see more below), I am not sure this would be the case for other
> > architectures.
> > The closest thing to MSR on Arm would be the System Registers. But I
> > am not aware of a range of IDs that could be used by the software.
>
> I don't really know that much about Arm to make any helpful statement
> here. My point was to keep in mind that it might be interesting to try
> to define an hypervisor agnostic mediated data exchange interface, so
> that whatever is implemented in the VirtIO transport layer could also
> be used by other hypervisors without having to modify the transport
> layer itself. If that's better done using a vmcall interface that's
> fine, as long as we provide a sane interface that other hypervisors
> can (easily) implement.
>
> > >
> > > > > IMO it might be easier to start by having an Argo interface using
> > > > > MSRs, that all hypervisors can implement, and then base the VirtIO
> > > > > implementation on top of that interface.
> > > > My concern is the interface would need to be arch-specific. Would you 
> > > > mind
> > > > to explain what's the problem to implement something based on vmcall?
> > >
> > > More of a recommendation than a concern, but I think it would be more
> > > attractive for upstream if we could provide an interface to Argo (or
> > > hypervisor mediated data exchange) that's no too tied into Xen
> > > specifics.
> >
> > I agree with this statement. We also need an interface that is ideally
> > not to arch-specific otherwise it will be more complicated to get
> > adopted on other arch.
> > For instance, at the moment, I don't really see what else can be used
> > on Arm (other that MMIO and PCI) if you want to care about PV (or
> > binary translated) guests.
>
> My recommendation was mostly to make Argo easier to propose as an
> hypervisor agnostic mediated data exchange, which in turn could make
> adding a VirtIO transport layer based on it easier. If that's not the
> case let's just forget about this.
>
> > > My suggestion for using MSRs was because I think every x86 hypervisor
> > > must have the logic to trap and handle some of those, and also every
> > > OS has the helpers to read/write MSRs, and those instructions are not
> > > vendor specific.
> >
> > In order to use MSRs, you would need to reserve a range of IDs. Do x86
> > have a range of ID that can be used for software purposes (i.e. the
> > current and future processors will not use it)?
>
> There's a range of MSRs (0x40000000-0x400000FF) that are guaranteed to
> always be invalid on bare metal by Intel, so VMware, HyperV and
> others have started using this range to add virtualization specific
> MSRs. You can grep for HV_X64_MSR_* defines on Xen for some of the
> HyperV ones.

I've written a summary of the points from this thread in a project
description at this wiki page:

https://openxt.atlassian.net/wiki/spaces/DC/pages/1696169985/VirtIO-Argo+Development+Phase+1#Project:-Hypervisor-agnostic-Hypervisor-Interface

-- please let me know if anything is captured incorrectly or any
amendments that you would like to be made.

As a reminder, we have an upcoming VirtIO-Argo call on Thursday next
week, 14th of January at 16:00 UTC.

thanks,

Christopher