RE: Enabling hypervisor agnosticism for VirtIO backends

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Akashi,

> -----Original Message-----
> From: AKASHI Takahiro <takahiro.akashi@xxxxxxxxxx>
> Sent: 2021年9月1日 20:29
> To: Wei Chen <Wei.Chen@xxxxxxx>
> Cc: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>; Stefano Stabellini
> <sstabellini@xxxxxxxxxx>; Alex Benn??e <alex.bennee@xxxxxxxxxx>; Kaly Xin
> <Kaly.Xin@xxxxxxx>; Stratos Mailing List <stratos-dev@xxxxxxxxxxxxxxxxxxx>;
> virtio-dev@xxxxxxxxxxxxxxxxxxxx; Arnd Bergmann <arnd.bergmann@xxxxxxxxxx>;
> Viresh Kumar <viresh.kumar@xxxxxxxxxx>; Stefano Stabellini
> <stefano.stabellini@xxxxxxxxxx>; stefanha@xxxxxxxxxx; Jan Kiszka
> <jan.kiszka@xxxxxxxxxxx>; Carl van Schaik <cvanscha@xxxxxxxxxxxxxxxx>;
> pratikp@xxxxxxxxxxx; Srivatsa Vaddagiri <vatsa@xxxxxxxxxxxxxx>; Jean-
> Philippe Brucker <jean-philippe@xxxxxxxxxx>; Mathieu Poirier
> <mathieu.poirier@xxxxxxxxxx>; Oleksandr Tyshchenko
> <Oleksandr_Tyshchenko@xxxxxxxx>; Bertrand Marquis
> <Bertrand.Marquis@xxxxxxx>; Artem Mygaiev <Artem_Mygaiev@xxxxxxxx>; Julien
> Grall <julien@xxxxxxx>; Juergen Gross <jgross@xxxxxxxx>; Paul Durrant
> <paul@xxxxxxx>; nd <nd@xxxxxxx>; Xen Devel <xen-devel@xxxxxxxxxxxxx>
> Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
> 
> Hi Wei,
> 
> On Wed, Sep 01, 2021 at 11:12:58AM +0000, Wei Chen wrote:
> > Hi Akashi,
> >
> > > -----Original Message-----
> > > From: AKASHI Takahiro <takahiro.akashi@xxxxxxxxxx>
> > > Sent: 2021年8月31日 14:18
> > > To: Wei Chen <Wei.Chen@xxxxxxx>
> > > Cc: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>; Stefano Stabellini
> > > <sstabellini@xxxxxxxxxx>; Alex Benn??e <alex.bennee@xxxxxxxxxx>; Kaly
> Xin
> > > <Kaly.Xin@xxxxxxx>; Stratos Mailing List <stratos-dev@op-
> lists.linaro.org>;
> > > virtio-dev@xxxxxxxxxxxxxxxxxxxx; Arnd Bergmann
> <arnd.bergmann@xxxxxxxxxx>;
> > > Viresh Kumar <viresh.kumar@xxxxxxxxxx>; Stefano Stabellini
> > > <stefano.stabellini@xxxxxxxxxx>; stefanha@xxxxxxxxxx; Jan Kiszka
> > > <jan.kiszka@xxxxxxxxxxx>; Carl van Schaik <cvanscha@xxxxxxxxxxxxxxxx>;
> > > pratikp@xxxxxxxxxxx; Srivatsa Vaddagiri <vatsa@xxxxxxxxxxxxxx>; Jean-
> > > Philippe Brucker <jean-philippe@xxxxxxxxxx>; Mathieu Poirier
> > > <mathieu.poirier@xxxxxxxxxx>; Oleksandr Tyshchenko
> > > <Oleksandr_Tyshchenko@xxxxxxxx>; Bertrand Marquis
> > > <Bertrand.Marquis@xxxxxxx>; Artem Mygaiev <Artem_Mygaiev@xxxxxxxx>;
> Julien
> > > Grall <julien@xxxxxxx>; Juergen Gross <jgross@xxxxxxxx>; Paul Durrant
> > > <paul@xxxxxxx>; Xen Devel <xen-devel@xxxxxxxxxxxxx>
> > > Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
> > >
> > > Wei,
> > >
> > > On Thu, Aug 26, 2021 at 12:10:19PM +0000, Wei Chen wrote:
> > > > Hi Akashi,
> > > >
> > > > > -----Original Message-----
> > > > > From: AKASHI Takahiro <takahiro.akashi@xxxxxxxxxx>
> > > > > Sent: 2021年8月26日 17:41
> > > > > To: Wei Chen <Wei.Chen@xxxxxxx>
> > > > > Cc: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>; Stefano Stabellini
> > > > > <sstabellini@xxxxxxxxxx>; Alex Benn??e <alex.bennee@xxxxxxxxxx>;
> Kaly
> > > Xin
> > > > > <Kaly.Xin@xxxxxxx>; Stratos Mailing List <stratos-dev@op-
> > > lists.linaro.org>;
> > > > > virtio-dev@xxxxxxxxxxxxxxxxxxxx; Arnd Bergmann
> > > <arnd.bergmann@xxxxxxxxxx>;
> > > > > Viresh Kumar <viresh.kumar@xxxxxxxxxx>; Stefano Stabellini
> > > > > <stefano.stabellini@xxxxxxxxxx>; stefanha@xxxxxxxxxx; Jan Kiszka
> > > > > <jan.kiszka@xxxxxxxxxxx>; Carl van Schaik
> <cvanscha@xxxxxxxxxxxxxxxx>;
> > > > > pratikp@xxxxxxxxxxx; Srivatsa Vaddagiri <vatsa@xxxxxxxxxxxxxx>;
> Jean-
> > > > > Philippe Brucker <jean-philippe@xxxxxxxxxx>; Mathieu Poirier
> > > > > <mathieu.poirier@xxxxxxxxxx>; Oleksandr Tyshchenko
> > > > > <Oleksandr_Tyshchenko@xxxxxxxx>; Bertrand Marquis
> > > > > <Bertrand.Marquis@xxxxxxx>; Artem Mygaiev <Artem_Mygaiev@xxxxxxxx>;
> > > Julien
> > > > > Grall <julien@xxxxxxx>; Juergen Gross <jgross@xxxxxxxx>; Paul
> Durrant
> > > > > <paul@xxxxxxx>; Xen Devel <xen-devel@xxxxxxxxxxxxx>
> > > > > Subject: Re: Enabling hypervisor agnosticism for VirtIO backends
> > > > >
> > > > > Hi Wei,
> > > > >
> > > > > On Fri, Aug 20, 2021 at 03:41:50PM +0900, AKASHI Takahiro wrote:
> > > > > > On Wed, Aug 18, 2021 at 08:35:51AM +0000, Wei Chen wrote:
> > > > > > > Hi Akashi，
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: AKASHI Takahiro <takahiro.akashi@xxxxxxxxxx>
> > > > > > > > Sent: 2021年8月18日 13:39
> > > > > > > > To: Wei Chen <Wei.Chen@xxxxxxx>
> > > > > > > > Cc: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>; Stefano
> > > Stabellini
> > > > > > > > <sstabellini@xxxxxxxxxx>; Alex Benn??e
> <alex.bennee@xxxxxxxxxx>;
> > > > > Stratos
> > > > > > > > Mailing List <stratos-dev@xxxxxxxxxxxxxxxxxxx>; virtio-
> > > > > dev@lists.oasis-
> > > > > > > > open.org; Arnd Bergmann <arnd.bergmann@xxxxxxxxxx>; Viresh
> Kumar
> > > > > > > > <viresh.kumar@xxxxxxxxxx>; Stefano Stabellini
> > > > > > > > <stefano.stabellini@xxxxxxxxxx>; stefanha@xxxxxxxxxx; Jan
> Kiszka
> > > > > > > > <jan.kiszka@xxxxxxxxxxx>; Carl van Schaik
> > > > > <cvanscha@xxxxxxxxxxxxxxxx>;
> > > > > > > > pratikp@xxxxxxxxxxx; Srivatsa Vaddagiri
> <vatsa@xxxxxxxxxxxxxx>;
> > > > > Jean-
> > > > > > > > Philippe Brucker <jean-philippe@xxxxxxxxxx>; Mathieu Poirier
> > > > > > > > <mathieu.poirier@xxxxxxxxxx>; Oleksandr Tyshchenko
> > > > > > > > <Oleksandr_Tyshchenko@xxxxxxxx>; Bertrand Marquis
> > > > > > > > <Bertrand.Marquis@xxxxxxx>; Artem Mygaiev
> > > <Artem_Mygaiev@xxxxxxxx>;
> > > > > Julien
> > > > > > > > Grall <julien@xxxxxxx>; Juergen Gross <jgross@xxxxxxxx>;
> Paul
> > > > > Durrant
> > > > > > > > <paul@xxxxxxx>; Xen Devel <xen-devel@xxxxxxxxxxxxx>
> > > > > > > > Subject: Re: Enabling hypervisor agnosticism for VirtIO
> backends
> > > > > > > >
> > > > > > > > On Tue, Aug 17, 2021 at 08:39:09AM +0000, Wei Chen wrote:
> > > > > > > > > Hi Akashi,
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: AKASHI Takahiro <takahiro.akashi@xxxxxxxxxx>
> > > > > > > > > > Sent: 2021年8月17日 16:08
> > > > > > > > > > To: Wei Chen <Wei.Chen@xxxxxxx>
> > > > > > > > > > Cc: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>; Stefano
> > > > > Stabellini
> > > > > > > > > > <sstabellini@xxxxxxxxxx>; Alex Benn??e
> > > <alex.bennee@xxxxxxxxxx>;
> > > > > > > > Stratos
> > > > > > > > > > Mailing List <stratos-dev@xxxxxxxxxxxxxxxxxxx>; virtio-
> > > > > > > > dev@lists.oasis-
> > > > > > > > > > open.org; Arnd Bergmann <arnd.bergmann@xxxxxxxxxx>;
> Viresh
> > > Kumar
> > > > > > > > > > <viresh.kumar@xxxxxxxxxx>; Stefano Stabellini
> > > > > > > > > > <stefano.stabellini@xxxxxxxxxx>; stefanha@xxxxxxxxxx;
> Jan
> > > Kiszka
> > > > > > > > > > <jan.kiszka@xxxxxxxxxxx>; Carl van Schaik
> > > > > <cvanscha@xxxxxxxxxxxxxxxx>;
> > > > > > > > > > pratikp@xxxxxxxxxxx; Srivatsa Vaddagiri
> > > <vatsa@xxxxxxxxxxxxxx>;
> > > > > Jean-
> > > > > > > > > > Philippe Brucker <jean-philippe@xxxxxxxxxx>; Mathieu
> Poirier
> > > > > > > > > > <mathieu.poirier@xxxxxxxxxx>; Oleksandr Tyshchenko
> > > > > > > > > > <Oleksandr_Tyshchenko@xxxxxxxx>; Bertrand Marquis
> > > > > > > > > > <Bertrand.Marquis@xxxxxxx>; Artem Mygaiev
> > > > > <Artem_Mygaiev@xxxxxxxx>;
> > > > > > > > Julien
> > > > > > > > > > Grall <julien@xxxxxxx>; Juergen Gross <jgross@xxxxxxxx>;
> > > Paul
> > > > > Durrant
> > > > > > > > > > <paul@xxxxxxx>; Xen Devel <xen-devel@xxxxxxxxxxxxx>
> > > > > > > > > > Subject: Re: Enabling hypervisor agnosticism for VirtIO
> > > backends
> > > > > > > > > >
> > > > > > > > > > Hi Wei, Oleksandr,
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 16, 2021 at 10:04:03AM +0000, Wei Chen wrote:
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for Stefano to link my kvmtool for Xen proposal
> > > here.
> > > > > > > > > > > This proposal is still discussing in Xen and KVM
> > > communities.
> > > > > > > > > > > The main work is to decouple the kvmtool from KVM and
> make
> > > > > > > > > > > other hypervisors can reuse the virtual device
> > > implementations.
> > > > > > > > > > >
> > > > > > > > > > > In this case, we need to introduce an intermediate
> > > hypervisor
> > > > > > > > > > > layer for VMM abstraction, Which is, I think it's very
> > > close
> > > > > > > > > > > to stratos' virtio hypervisor agnosticism work.
> > > > > > > > > >
> > > > > > > > > > # My proposal[1] comes from my own idea and doesn't
> always
> > > > > represent
> > > > > > > > > > # Linaro's view on this subject nor reflect Alex's
> concerns.
> > > > > > > > Nevertheless,
> > > > > > > > > >
> > > > > > > > > > Your idea and my proposal seem to share the same
> background.
> > > > > > > > > > Both have the similar goal and currently start with, at
> > > first,
> > > > > Xen
> > > > > > > > > > and are based on kvm-tool. (Actually, my work is derived
> > > from
> > > > > > > > > > EPAM's virtio-disk, which is also based on kvm-tool.)
> > > > > > > > > >
> > > > > > > > > > In particular, the abstraction of hypervisor interfaces
> has
> > > a
> > > > > same
> > > > > > > > > > set of interfaces (for your "struct vmm_impl" and my
> "RPC
> > > > > interfaces").
> > > > > > > > > > This is not co-incident as we both share the same origin
> as
> > > I
> > > > > said
> > > > > > > > above.
> > > > > > > > > > And so we will also share the same issues. One of them
> is a
> > > way
> > > > > of
> > > > > > > > > > "sharing/mapping FE's memory". There is some trade-off
> > > between
> > > > > > > > > > the portability and the performance impact.
> > > > > > > > > > So we can discuss the topic here in this ML, too.
> > > > > > > > > > (See Alex's original email, too).
> > > > > > > > > >
> > > > > > > > > Yes, I agree.
> > > > > > > > >
> > > > > > > > > > On the other hand, my approach aims to create a "single-
> > > binary"
> > > > > > > > solution
> > > > > > > > > > in which the same binary of BE vm could run on any
> > > hypervisors.
> > > > > > > > > > Somehow similar to your "proposal-#2" in [2], but in my
> > > solution,
> > > > > all
> > > > > > > > > > the hypervisor-specific code would be put into another
> > > entity
> > > > > (VM),
> > > > > > > > > > named "virtio-proxy" and the abstracted operations are
> > > served
> > > > > via RPC.
> > > > > > > > > > (In this sense, BE is hypervisor-agnostic but might have
> OS
> > > > > > > > dependency.)
> > > > > > > > > > But I know that we need discuss if this is a requirement
> > > even
> > > > > > > > > > in Stratos project or not. (Maybe not)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Sorry, I haven't had time to finish reading your virtio-
> proxy
> > > > > completely
> > > > > > > > > (I will do it ASAP). But from your description, it seems
> we
> > > need a
> > > > > > > > > 3rd VM between FE and BE? My concern is that, if my
> assumption
> > > is
> > > > > right,
> > > > > > > > > will it increase the latency in data transport path? Even
> if
> > > we're
> > > > > > > > > using some lightweight guest like RTOS or Unikernel,
> > > > > > > >
> > > > > > > > Yes, you're right. But I'm afraid that it is a matter of
> degree.
> > > > > > > > As far as we execute 'mapping' operations at every fetch of
> > > payload,
> > > > > > > > we will see latency issue (even in your case) and if we have
> > > some
> > > > > solution
> > > > > > > > for it, we won't see it neither in my proposal :)
> > > > > > > >
> > > > > > >
> > > > > > > Oleksandr has sent a proposal to Xen mailing list to reduce
> this
> > > kind
> > > > > > > of "mapping/unmapping" operations. So the latency caused by
> this
> > > > > behavior
> > > > > > > on Xen may eventually be eliminated, and Linux-KVM doesn't
> have
> > > that
> > > > > problem.
> > > > > >
> > > > > > Obviously, I have not yet caught up there in the discussion.
> > > > > > Which patch specifically?
> > > > >
> > > > > Can you give me the link to the discussion or patch, please?
> > > > >
> > > >
> > > > It's a RFC discussion. We have tested this RFC patch internally.
> > > > https://lists.xenproject.org/archives/html/xen-devel/2021-
> > > 07/msg01532.html
> > >
> > > I'm afraid that I miss something here, but I don't know
> > > why this proposed API will lead to eliminating 'mmap' in accessing
> > > the queued payload at every request?
> > >
> >
> > This API give Xen device model (QEMU or kvmtool) the ability to map
> > whole guest RAM in device model's address space. In this case, device
> > model doesn't need dynamic hypercall to map/unmap payload memory.
> > It can use a flat offset to access payload memory in its address
> > space directly. Just Like KVM device model does now.
> 
> Thank you. Quickly, let me make sure one thing:
> This API itself doesn't do any mapping operations, right?
> So I suppose that virtio BE guest is responsible to
> 1) fetch the information about all the memory regions in FE,
> 2) call this API to allocate a big chunk of unused space in BE,
> 3) create grant/foreign mappings for FE onto this region(S)
> in the initialization/configuration of emulated virtio devices.
> 
> Is this the way this API is expected to be used?
> Does Xen already has an interface for (1)?
> 

They are discussing in that thread to find a proper way to do it.
Because this API is common, both x86 and Arm should be considered.

> -Takahiro Akashi
> 
> > Before this API, When device model to map whole guest memory, will
> > severely consume the physical pages of Dom-0/Dom-D.
> >
> > > -Takahiro Akashi
> > >
> > >
> > > > > Thanks,
> > > > > -Takahiro Akashi
> > > > >
> > > > > > -Takahiro Akashi
> > > > > >
> > > > > > > > > > Specifically speaking about kvm-tool, I have a concern
> about
> > > its
> > > > > > > > > > license term; Targeting different hypervisors and
> different
> > > OSs
> > > > > > > > > > (which I assume includes RTOS's), the resultant library
> > > should
> > > > > be
> > > > > > > > > > license permissive and GPL for kvm-tool might be an
> issue.
> > > > > > > > > > Any thoughts?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes. If user want to implement a FreeBSD device model, but
> the
> > > > > virtio
> > > > > > > > > library is GPL. Then GPL would be a problem. If we have
> > > another
> > > > > good
> > > > > > > > > candidate, I am open to it.
> > > > > > > >
> > > > > > > > I have some candidates, particularly for vq/vring, in my
> mind:
> > > > > > > > * Open-AMP, or
> > > > > > > > * corresponding Free-BSD code
> > > > > > > >
> > > > > > >
> > > > > > > Interesting, I will look into them : )
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Wei Chen
> > > > > > >
> > > > > > > > -Takahiro Akashi
> > > > > > > >
> > > > > > > >
> > > > > > > > > > -Takahiro Akashi
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1] https://op-lists.linaro.org/pipermail/stratos-
> dev/2021-
> > > > > > > > > > August/000548.html
> > > > > > > > > > [2] https://marc.info/?l=xen-devel&m=162373754705233&w=2
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > From: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>
> > > > > > > > > > > > Sent: 2021年8月14日 23:38
> > > > > > > > > > > > To: AKASHI Takahiro <takahiro.akashi@xxxxxxxxxx>;
> > > Stefano
> > > > > > > > Stabellini
> > > > > > > > > > <sstabellini@xxxxxxxxxx>
> > > > > > > > > > > > Cc: Alex Benn??e <alex.bennee@xxxxxxxxxx>; Stratos
> > > Mailing
> > > > > List
> > > > > > > > > > <stratos-dev@xxxxxxxxxxxxxxxxxxx>; virtio-
> dev@lists.oasis-
> > > > > open.org;
> > > > > > > > Arnd
> > > > > > > > > > Bergmann <arnd.bergmann@xxxxxxxxxx>; Viresh Kumar
> > > > > > > > > > <viresh.kumar@xxxxxxxxxx>; Stefano Stabellini
> > > > > > > > > > <stefano.stabellini@xxxxxxxxxx>; stefanha@xxxxxxxxxx;
> Jan
> > > Kiszka
> > > > > > > > > > <jan.kiszka@xxxxxxxxxxx>; Carl van Schaik
> > > > > <cvanscha@xxxxxxxxxxxxxxxx>;
> > > > > > > > > > pratikp@xxxxxxxxxxx; Srivatsa Vaddagiri
> > > <vatsa@xxxxxxxxxxxxxx>;
> > > > > Jean-
> > > > > > > > > > Philippe Brucker <jean-philippe@xxxxxxxxxx>; Mathieu
> Poirier
> > > > > > > > > > <mathieu.poirier@xxxxxxxxxx>; Wei Chen
> <Wei.Chen@xxxxxxx>;
> > > > > Oleksandr
> > > > > > > > > > Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>; Bertrand
> Marquis
> > > > > > > > > > <Bertrand.Marquis@xxxxxxx>; Artem Mygaiev
> > > > > <Artem_Mygaiev@xxxxxxxx>;
> > > > > > > > Julien
> > > > > > > > > > Grall <julien@xxxxxxx>; Juergen Gross <jgross@xxxxxxxx>;
> > > Paul
> > > > > Durrant
> > > > > > > > > > <paul@xxxxxxx>; Xen Devel <xen-devel@xxxxxxxxxxxxx>
> > > > > > > > > > > > Subject: Re: Enabling hypervisor agnosticism for
> VirtIO
> > > > > backends
> > > > > > > > > > > >
> > > > > > > > > > > > Hello, all.
> > > > > > > > > > > >
> > > > > > > > > > > > Please see some comments below. And sorry for the
> > > possible
> > > > > format
> > > > > > > > > > issues.
> > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Aug 11, 2021 at 9:27 AM AKASHI Takahiro
> > > > > > > > > > <mailto:takahiro.akashi@xxxxxxxxxx> wrote:
> > > > > > > > > > > > > On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano
> > > > > Stabellini
> > > > > > > > wrote:
> > > > > > > > > > > > > > CCing people working on Xen+VirtIO and IOREQs.
> Not
> > > > > trimming
> > > > > > > > the
> > > > > > > > > > original
> > > > > > > > > > > > > > email to let them read the full context.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > My comments below are related to a potential Xen
> > > > > > > > implementation,
> > > > > > > > > > not
> > > > > > > > > > > > > > because it is the only implementation that
> matters,
> > > but
> > > > > > > > because it
> > > > > > > > > > is
> > > > > > > > > > > > > > the one I know best.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please note that my proposal (and hence the
> working
> > > > > prototype)[1]
> > > > > > > > > > > > > is based on Xen's virtio implementation (i.e.
> IOREQ)
> > > and
> > > > > > > > > > particularly
> > > > > > > > > > > > > EPAM's virtio-disk application (backend server).
> > > > > > > > > > > > > It has been, I believe, well generalized but is
> still
> > > a
> > > > > bit
> > > > > > > > biased
> > > > > > > > > > > > > toward this original design.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So I hope you like my approach :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1] https://op-lists.linaro.org/pipermail/stratos-
> > > > > dev/2021-
> > > > > > > > > > August/000546.html
> > > > > > > > > > > > >
> > > > > > > > > > > > > Let me take this opportunity to explain a bit more
> > > about
> > > > > my
> > > > > > > > approach
> > > > > > > > > > below.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Also, please see this relevant email thread:
> > > > > > > > > > > > > > https://marc.info/?l=xen-
> devel&m=162373754705233&w=2
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, 4 Aug 2021, Alex Bennée wrote:
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One of the goals of Project Stratos is to
> enable
> > > > > hypervisor
> > > > > > > > > > agnostic
> > > > > > > > > > > > > > > backends so we can enable as much re-use of
> code
> > > as
> > > > > possible
> > > > > > > > and
> > > > > > > > > > avoid
> > > > > > > > > > > > > > > repeating ourselves. This is the flip side of
> the
> > > > > front end
> > > > > > > > > > where
> > > > > > > > > > > > > > > multiple front-end implementations are
> required -
> > > one
> > > > > per OS,
> > > > > > > > > > assuming
> > > > > > > > > > > > > > > you don't just want Linux guests. The
> resultant
> > > guests
> > > > > are
> > > > > > > > > > trivially
> > > > > > > > > > > > > > > movable between hypervisors modulo any
> abstracted
> > > > > paravirt
> > > > > > > > type
> > > > > > > > > > > > > > > interfaces.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In my original thumb nail sketch of a solution
> I
> > > > > envisioned
> > > > > > > > > > vhost-user
> > > > > > > > > > > > > > > daemons running in a broadly POSIX like
> > > environment.
> > > > > The
> > > > > > > > > > interface to
> > > > > > > > > > > > > > > the daemon is fairly simple requiring only
> some
> > > mapped
> > > > > > > > memory
> > > > > > > > > > and some
> > > > > > > > > > > > > > > sort of signalling for events (on Linux this
> is
> > > > > eventfd).
> > > > > > > > The
> > > > > > > > > > idea was a
> > > > > > > > > > > > > > > stub binary would be responsible for any
> > > hypervisor
> > > > > specific
> > > > > > > > > > setup and
> > > > > > > > > > > > > > > then launch a common binary to deal with the
> > > actual
> > > > > > > > virtqueue
> > > > > > > > > > requests
> > > > > > > > > > > > > > > themselves.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Since that original sketch we've seen an
> expansion
> > > in
> > > > > the
> > > > > > > > sort
> > > > > > > > > > of ways
> > > > > > > > > > > > > > > backends could be created. There is interest
> in
> > > > > > > > encapsulating
> > > > > > > > > > backends
> > > > > > > > > > > > > > > in RTOSes or unikernels for solutions like
> SCMI.
> > > There
> > > > > > > > interest
> > > > > > > > > > in Rust
> > > > > > > > > > > > > > > has prompted ideas of using the trait
> interface to
> > > > > abstract
> > > > > > > > > > differences
> > > > > > > > > > > > > > > away as well as the idea of bare-metal Rust
> > > backends.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We have a card (STR-12) called "Hypercall
> > > > > Standardisation"
> > > > > > > > which
> > > > > > > > > > > > > > > calls for a description of the APIs needed
> from
> > > the
> > > > > > > > hypervisor
> > > > > > > > > > side to
> > > > > > > > > > > > > > > support VirtIO guests and their backends.
> However
> > > we
> > > > > are
> > > > > > > > some
> > > > > > > > > > way off
> > > > > > > > > > > > > > > from that at the moment as I think we need to
> at
> > > least
> > > > > > > > > > demonstrate one
> > > > > > > > > > > > > > > portable backend before we start codifying
> > > > > requirements. To
> > > > > > > > that
> > > > > > > > > > end I
> > > > > > > > > > > > > > > want to think about what we need for a backend
> to
> > > > > function.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Configuration
> > > > > > > > > > > > > > > =============
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the type-2 setup this is typically fairly
> > > simple
> > > > > because
> > > > > > > > the
> > > > > > > > > > host
> > > > > > > > > > > > > > > system can orchestrate the various modules
> that
> > > make
> > > > > up the
> > > > > > > > > > complete
> > > > > > > > > > > > > > > system. In the type-1 case (or even type-2
> with
> > > > > delegated
> > > > > > > > > > service VMs)
> > > > > > > > > > > > > > > we need some sort of mechanism to inform the
> > > backend
> > > > > VM
> > > > > > > > about
> > > > > > > > > > key
> > > > > > > > > > > > > > > details about the system:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >   - where virt queue memory is in it's address
> > > space
> > > > > > > > > > > > > > >   - how it's going to receive (interrupt) and
> > > trigger
> > > > > (kick)
> > > > > > > > > > events
> > > > > > > > > > > > > > >   - what (if any) resources the backend needs
> to
> > > > > connect to
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Obviously you can elide over configuration
> issues
> > > by
> > > > > having
> > > > > > > > > > static
> > > > > > > > > > > > > > > configurations and baking the assumptions into
> > > your
> > > > > guest
> > > > > > > > images
> > > > > > > > > > however
> > > > > > > > > > > > > > > this isn't scalable in the long term. The
> obvious
> > > > > solution
> > > > > > > > seems
> > > > > > > > > > to be
> > > > > > > > > > > > > > > extending a subset of Device Tree data to user
> > > space
> > > > > but
> > > > > > > > perhaps
> > > > > > > > > > there
> > > > > > > > > > > > > > > are other approaches?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Before any virtio transactions can take place
> the
> > > > > > > > appropriate
> > > > > > > > > > memory
> > > > > > > > > > > > > > > mappings need to be made between the FE guest
> and
> > > the
> > > > > BE
> > > > > > > > guest.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently the whole of the FE guests address
> space
> > > > > needs to
> > > > > > > > be
> > > > > > > > > > visible
> > > > > > > > > > > > > > > to whatever is serving the virtio requests. I
> can
> > > > > envision 3
> > > > > > > > > > approaches:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  * BE guest boots with memory already mapped
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  This would entail the guest OS knowing where
> in
> > > it's
> > > > > Guest
> > > > > > > > > > Physical
> > > > > > > > > > > > > > >  Address space is already taken up and
> avoiding
> > > > > clashing. I
> > > > > > > > > > would assume
> > > > > > > > > > > > > > >  in this case you would want a standard
> interface
> > > to
> > > > > > > > userspace
> > > > > > > > > > to then
> > > > > > > > > > > > > > >  make that address space visible to the
> backend
> > > daemon.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yet another way here is that we would have well
> known
> > > > > "shared
> > > > > > > > > > memory" between
> > > > > > > > > > > > > VMs. I think that Jailhouse's ivshmem gives us
> good
> > > > > insights on
> > > > > > > > this
> > > > > > > > > > matter
> > > > > > > > > > > > > and that it can even be an alternative for
> hypervisor-
> > > > > agnostic
> > > > > > > > > > solution.
> > > > > > > > > > > > >
> > > > > > > > > > > > > (Please note memory regions in ivshmem appear as a
> PCI
> > > > > device
> > > > > > > > and
> > > > > > > > > > can be
> > > > > > > > > > > > > mapped locally.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > I want to add this shared memory aspect to my
> virtio-
> > > proxy,
> > > > > but
> > > > > > > > > > > > > the resultant solution would eventually look
> similar
> > > to
> > > > > ivshmem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > >  * BE guests boots with a hypervisor handle to
> > > memory
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  The BE guest is then free to map the FE's
> memory
> > > to
> > > > > where
> > > > > > > > it
> > > > > > > > > > wants in
> > > > > > > > > > > > > > >  the BE's guest physical address space.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I cannot see how this could work for Xen. There
> is
> > > no
> > > > > "handle"
> > > > > > > > to
> > > > > > > > > > give
> > > > > > > > > > > > > > to the backend if the backend is not running in
> dom0.
> > > So
> > > > > for
> > > > > > > > Xen I
> > > > > > > > > > think
> > > > > > > > > > > > > > the memory has to be already mapped
> > > > > > > > > > > > >
> > > > > > > > > > > > > In Xen's IOREQ solution (virtio-blk), the
> following
> > > > > information
> > > > > > > > is
> > > > > > > > > > expected
> > > > > > > > > > > > > to be exposed to BE via Xenstore:
> > > > > > > > > > > > > (I know that this is a tentative approach though.)
> > > > > > > > > > > > >    - the start address of configuration space
> > > > > > > > > > > > >    - interrupt number
> > > > > > > > > > > > >    - file path for backing storage
> > > > > > > > > > > > >    - read-only flag
> > > > > > > > > > > > > And the BE server have to call a particular
> hypervisor
> > > > > interface
> > > > > > > > to
> > > > > > > > > > > > > map the configuration space.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, Xenstore was chosen as a simple way to pass
> > > > > configuration
> > > > > > > > info to
> > > > > > > > > > the backend running in a non-toolstack domain.
> > > > > > > > > > > > I remember, there was a wish to avoid using Xenstore
> in
> > > > > Virtio
> > > > > > > > backend
> > > > > > > > > > itself if possible, so for non-toolstack domain, this
> could
> > > done
> > > > > with
> > > > > > > > > > adjusting devd (daemon that listens for devices and
> launches
> > > > > backends)
> > > > > > > > > > > > to read backend configuration from the Xenstore
> anyway
> > > and
> > > > > pass it
> > > > > > > > to
> > > > > > > > > > the backend via command line arguments.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, in current PoC code we're using xenstore to pass
> > > device
> > > > > > > > > > configuration.
> > > > > > > > > > > We also designed a static device configuration parse
> > > method
> > > > > for
> > > > > > > > Dom0less
> > > > > > > > > > or
> > > > > > > > > > > other scenarios don't have xentool. yes, it's from
> device
> > > > > model
> > > > > > > > command
> > > > > > > > > > line
> > > > > > > > > > > or a config file.
> > > > > > > > > > >
> > > > > > > > > > > > But, if ...
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > In my approach (virtio-proxy), all those Xen (or
> > > > > hypervisor)-
> > > > > > > > > > specific
> > > > > > > > > > > > > stuffs are contained in virtio-proxy, yet another
> VM,
> > > to
> > > > > hide
> > > > > > > > all
> > > > > > > > > > details.
> > > > > > > > > > > >
> > > > > > > > > > > > ... the solution how to overcome that is already
> found
> > > and
> > > > > proven
> > > > > > > > to
> > > > > > > > > > work then even better.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > # My point is that a "handle" is not mandatory for
> > > > > executing
> > > > > > > > mapping.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > and the mapping probably done by the
> > > > > > > > > > > > > > toolstack (also see below.) Or we would have to
> > > invent a
> > > > > new
> > > > > > > > Xen
> > > > > > > > > > > > > > hypervisor interface and Xen virtual machine
> > > privileges
> > > > > to
> > > > > > > > allow
> > > > > > > > > > this
> > > > > > > > > > > > > > kind of mapping.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > If we run the backend in Dom0 that we have no
> > > problems
> > > > > of
> > > > > > > > course.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One of difficulties on Xen that I found in my
> approach
> > > is
> > > > > that
> > > > > > > > > > calling
> > > > > > > > > > > > > such hypervisor intefaces (registering IOREQ,
> mapping
> > > > > memory) is
> > > > > > > > > > only
> > > > > > > > > > > > > allowed on BE servers themselvies and so we will
> have
> > > to
> > > > > extend
> > > > > > > > > > those
> > > > > > > > > > > > > interfaces.
> > > > > > > > > > > > > This, however, will raise some concern on security
> and
> > > > > privilege
> > > > > > > > > > distribution
> > > > > > > > > > > > > as Stefan suggested.
> > > > > > > > > > > >
> > > > > > > > > > > > We also faced policy related issues with Virtio
> backend
> > > > > running in
> > > > > > > > > > other than Dom0 domain in a "dummy" xsm mode. In our
> target
> > > > > system we
> > > > > > > > run
> > > > > > > > > > the backend in a driver
> > > > > > > > > > > > domain (we call it DomD) where the underlying H/W
> > > resides.
> > > > > We
> > > > > > > > trust it,
> > > > > > > > > > so we wrote policy rules (to be used in "flask" xsm mode)
> to
> > > > > provide
> > > > > > > > it
> > > > > > > > > > with a little bit more privileges than a simple DomU had.
> > > > > > > > > > > > Now it is permitted to issue device-model, resource
> and
> > > > > memory
> > > > > > > > > > mappings, etc calls.
> > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > To activate the mapping will
> > > > > > > > > > > > > > >  require some sort of hypercall to the
> hypervisor.
> > > I
> > > > > can see
> > > > > > > > two
> > > > > > > > > > options
> > > > > > > > > > > > > > >  at this point:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >   - expose the handle to userspace for
> > > daemon/helper
> > > > > to
> > > > > > > > trigger
> > > > > > > > > > the
> > > > > > > > > > > > > > >     mapping via existing hypercall interfaces.
> If
> > > > > using a
> > > > > > > > helper
> > > > > > > > > > you
> > > > > > > > > > > > > > >     would have a hypervisor specific one to
> avoid
> > > the
> > > > > daemon
> > > > > > > > > > having to
> > > > > > > > > > > > > > >     care too much about the details or push
> that
> > > > > complexity
> > > > > > > > into
> > > > > > > > > > a
> > > > > > > > > > > > > > >     compile time option for the daemon which
> would
> > > > > result in
> > > > > > > > > > different
> > > > > > > > > > > > > > >     binaries although a common source base.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >   - expose a new kernel ABI to abstract the
> > > hypercall
> > > > > > > > > > differences away
> > > > > > > > > > > > > > >     in the guest kernel. In this case the
> > > userspace
> > > > > would
> > > > > > > > > > essentially
> > > > > > > > > > > > > > >     ask for an abstract "map guest N memory to
> > > > > userspace
> > > > > > > > ptr"
> > > > > > > > > > and let
> > > > > > > > > > > > > > >     the kernel deal with the different
> hypercall
> > > > > interfaces.
> > > > > > > > > > This of
> > > > > > > > > > > > > > >     course assumes the majority of BE guests
> would
> > > be
> > > > > Linux
> > > > > > > > > > kernels and
> > > > > > > > > > > > > > >     leaves the bare-metal/unikernel approaches
> to
> > > > > their own
> > > > > > > > > > devices.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Operation
> > > > > > > > > > > > > > > =========
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The core of the operation of VirtIO is fairly
> > > simple.
> > > > > Once
> > > > > > > > the
> > > > > > > > > > > > > > > vhost-user feature negotiation is done it's a
> case
> > > of
> > > > > > > > receiving
> > > > > > > > > > update
> > > > > > > > > > > > > > > events and parsing the resultant virt queue
> for
> > > data.
> > > > > The
> > > > > > > > vhost-
> > > > > > > > > > user
> > > > > > > > > > > > > > > specification handles a bunch of setup before
> that
> > > > > point,
> > > > > > > > mostly
> > > > > > > > > > to
> > > > > > > > > > > > > > > detail where the virt queues are set up FD's
> for
> > > > > memory and
> > > > > > > > > > event
> > > > > > > > > > > > > > > communication. This is where the envisioned
> stub
> > > > > process
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > > > > responsible for getting the daemon up and
> ready to
> > > run.
> > > > > This
> > > > > > > > is
> > > > > > > > > > > > > > > currently done inside a big VMM like QEMU but
> I
> > > > > suspect a
> > > > > > > > modern
> > > > > > > > > > > > > > > approach would be to use the rust-vmm vhost
> crate.
> > > It
> > > > > would
> > > > > > > > then
> > > > > > > > > > either
> > > > > > > > > > > > > > > communicate with the kernel's abstracted ABI
> or be
> > > re-
> > > > > > > > targeted
> > > > > > > > > > as a
> > > > > > > > > > > > > > > build option for the various hypervisors.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One thing I mentioned before to Alex is that Xen
> > > doesn't
> > > > > have
> > > > > > > > VMMs
> > > > > > > > > > the
> > > > > > > > > > > > > > way they are typically envisioned and described
> in
> > > other
> > > > > > > > > > environments.
> > > > > > > > > > > > > > Instead, Xen has IOREQ servers. Each of them
> > > connects
> > > > > > > > > > independently to
> > > > > > > > > > > > > > Xen via the IOREQ interface. E.g. today multiple
> > > QEMUs
> > > > > could
> > > > > > > > be
> > > > > > > > > > used as
> > > > > > > > > > > > > > emulators for a single Xen VM, each of them
> > > connecting
> > > > > to Xen
> > > > > > > > > > > > > > independently via the IOREQ interface.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The component responsible for starting a daemon
> > > and/or
> > > > > setting
> > > > > > > > up
> > > > > > > > > > shared
> > > > > > > > > > > > > > interfaces is the toolstack: the xl command and
> the
> > > > > > > > libxl/libxc
> > > > > > > > > > > > > > libraries.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think that VM configuration management (or
> > > orchestration
> > > > > in
> > > > > > > > > > Startos
> > > > > > > > > > > > > jargon?) is a subject to debate in parallel.
> > > > > > > > > > > > > Otherwise, is there any good assumption to avoid
> it
> > > right
> > > > > now?
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Oleksandr and others I CCed have been working on
> > > ways
> > > > > for the
> > > > > > > > > > toolstack
> > > > > > > > > > > > > > to create virtio backends and setup memory
> mappings.
> > > > > They
> > > > > > > > might be
> > > > > > > > > > able
> > > > > > > > > > > > > > to provide more info on the subject. I do think
> we
> > > miss
> > > > > a way
> > > > > > > > to
> > > > > > > > > > provide
> > > > > > > > > > > > > > the configuration to the backend and anything
> else
> > > that
> > > > > the
> > > > > > > > > > backend
> > > > > > > > > > > > > > might require to start doing its job.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, some work has been done for the toolstack to
> handle
> > > > > Virtio
> > > > > > > > MMIO
> > > > > > > > > > devices in
> > > > > > > > > > > > general and Virtio block devices in particular.
> However,
> > > it
> > > > > has
> > > > > > > > not
> > > > > > > > > > been upstreaned yet.
> > > > > > > > > > > > Updated patches on review now:
> > > > > > > > > > > > https://lore.kernel.org/xen-devel/1621626361-29076-
> 1-
> > > git-
> > > > > send-
> > > > > > > > email-
> > > > > > > > > > olekstysh@xxxxxxxxx/
> > > > > > > > > > > >
> > > > > > > > > > > > There is an additional (also important) activity to
> > > > > improve/fix
> > > > > > > > > > foreign memory mapping on Arm which I am also involved
> in.
> > > > > > > > > > > > The foreign memory mapping is proposed to be used
> for
> > > Virtio
> > > > > > > > backends
> > > > > > > > > > (device emulators) if there is a need to run guest OS
> > > completely
> > > > > > > > > > unmodified.
> > > > > > > > > > > > Of course, the more secure way would be to use grant
> > > memory
> > > > > > > > mapping.
> > > > > > > > > > Brietly, the main difference between them is that with
> > > foreign
> > > > > mapping
> > > > > > > > the
> > > > > > > > > > backend
> > > > > > > > > > > > can map any guest memory it wants to map, but with
> grant
> > > > > mapping
> > > > > > > > it is
> > > > > > > > > > allowed to map only what was previously granted by the
> > > frontend.
> > > > > > > > > > > >
> > > > > > > > > > > > So, there might be a problem if we want to pre-map
> some
> > > > > guest
> > > > > > > > memory
> > > > > > > > > > in advance or to cache mappings in the backend in order
> to
> > > > > improve
> > > > > > > > > > performance (because the mapping/unmapping guest pages
> every
> > > > > request
> > > > > > > > > > requires a lot of back and forth to Xen + P2M updates).
> In a
> > > > > nutshell,
> > > > > > > > > > currently, in order to map a guest page into the backend
> > > address
> > > > > space
> > > > > > > > we
> > > > > > > > > > need to steal a real physical page from the backend
> domain.
> > > So,
> > > > > with
> > > > > > > > the
> > > > > > > > > > said optimizations we might end up with no free memory
> in
> > > the
> > > > > backend
> > > > > > > > > > domain (see XSA-300). And what we try to achieve is to
> not
> > > waste
> > > > > a
> > > > > > > > real
> > > > > > > > > > domain memory at all by providing safe non-allocated-yet
> (so
> > > > > unused)
> > > > > > > > > > address space for the foreign (and grant) pages to be
> mapped
> > > > > into,
> > > > > > > > this
> > > > > > > > > > enabling work implies Xen and Linux (and likely DTB
> bindings)
> > > > > changes.
> > > > > > > > > > However, as it turned out, for this to work in a proper
> and
> > > safe
> > > > > way
> > > > > > > > some
> > > > > > > > > > prereq work needs to be done.
> > > > > > > > > > > > You can find the related Xen discussion at:
> > > > > > > > > > > > https://lore.kernel.org/xen-devel/1627489110-25633-
> 1-
> > > git-
> > > > > send-
> > > > > > > > email-
> > > > > > > > > > olekstysh@xxxxxxxxx/
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One question is how to best handle
> notification
> > > and
> > > > > kicks.
> > > > > > > > The
> > > > > > > > > > existing
> > > > > > > > > > > > > > > vhost-user framework uses eventfd to signal
> the
> > > daemon
> > > > > > > > (although
> > > > > > > > > > QEMU
> > > > > > > > > > > > > > > is quite capable of simulating them when you
> use
> > > TCG).
> > > > > Xen
> > > > > > > > has
> > > > > > > > > > it's own
> > > > > > > > > > > > > > > IOREQ mechanism. However latency is an
> important
> > > > > factor and
> > > > > > > > > > having
> > > > > > > > > > > > > > > events go through the stub would add quite a
> lot.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yeah I think, regardless of anything else, we
> want
> > > the
> > > > > > > > backends to
> > > > > > > > > > > > > > connect directly to the Xen hypervisor.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In my approach,
> > > > > > > > > > > > >  a) BE -> FE: interrupts triggered by BE calling a
> > > > > hypervisor
> > > > > > > > > > interface
> > > > > > > > > > > > >               via virtio-proxy
> > > > > > > > > > > > >  b) FE -> BE: MMIO to config raises events (in
> event
> > > > > channels),
> > > > > > > > > > which is
> > > > > > > > > > > > >               converted to a callback to BE via
> > > virtio-
> > > > > proxy
> > > > > > > > > > > > >               (Xen's event channel is internnally
> > > > > implemented by
> > > > > > > > > > interrupts.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't know what "connect directly" means here,
> but
> > > > > sending
> > > > > > > > > > interrupts
> > > > > > > > > > > > > to the opposite side would be best efficient.
> > > > > > > > > > > > > Ivshmem, I suppose, takes this approach by
> utilizing
> > > PCI's
> > > > > msi-x
> > > > > > > > > > mechanism.
> > > > > > > > > > > >
> > > > > > > > > > > > Agree that MSI would be more efficient than SPI...
> > > > > > > > > > > > At the moment, in order to notify the frontend, the
> > > backend
> > > > > issues
> > > > > > > > a
> > > > > > > > > > specific device-model call to query Xen to inject a
> > > > > corresponding SPI
> > > > > > > > to
> > > > > > > > > > the guest.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Could we consider the kernel internally
> converting
> > > > > IOREQ
> > > > > > > > > > messages from
> > > > > > > > > > > > > > > the Xen hypervisor to eventfd events? Would
> this
> > > scale
> > > > > with
> > > > > > > > > > other kernel
> > > > > > > > > > > > > > > hypercall interfaces?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So any thoughts on what directions are worth
> > > > > experimenting
> > > > > > > > with?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One option we should consider is for each
> backend to
> > > > > connect
> > > > > > > > to
> > > > > > > > > > Xen via
> > > > > > > > > > > > > > the IOREQ interface. We could generalize the
> IOREQ
> > > > > interface
> > > > > > > > and
> > > > > > > > > > make it
> > > > > > > > > > > > > > hypervisor agnostic. The interface is really
> trivial
> > > and
> > > > > easy
> > > > > > > > to
> > > > > > > > > > add.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As I said above, my proposal does the same thing
> that
> > > you
> > > > > > > > mentioned
> > > > > > > > > > here :)
> > > > > > > > > > > > > The difference is that I do call hypervisor
> interfaces
> > > via
> > > > > > > > virtio-
> > > > > > > > > > proxy.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The only Xen-specific part is the notification
> > > mechanism,
> > > > > > > > which is
> > > > > > > > > > an
> > > > > > > > > > > > > > event channel. If we replaced the event channel
> with
> > > > > something
> > > > > > > > > > else the
> > > > > > > > > > > > > > interface would be generic. See:
> > > > > > > > > > > > > > https://gitlab.com/xen-project/xen/-
> > > > > > > > > > /blob/staging/xen/include/public/hvm/ioreq.h#L52
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't think that translating IOREQs to eventfd
> in
> > > the
> > > > > kernel
> > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > > > > good idea: if feels like it would be extra
> > > complexity
> > > > > and that
> > > > > > > > the
> > > > > > > > > > > > > > kernel shouldn't be involved as this is a
> backend-
> > > > > hypervisor
> > > > > > > > > > interface.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Given that we may want to implement BE as a bare-
> metal
> > > > > > > > application
> > > > > > > > > > > > > as I did on Zephyr, I don't think that the
> translation
> > > > > would not
> > > > > > > > be
> > > > > > > > > > > > > a big issue, especially on RTOS's.
> > > > > > > > > > > > > It will be some kind of abstraction layer of
> interrupt
> > > > > handling
> > > > > > > > > > > > > (or nothing but a callback mechanism).
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Also, eventfd is very Linux-centric and we are
> > > trying to
> > > > > > > > design an
> > > > > > > > > > > > > > interface that could work well for RTOSes too.
> If we
> > > > > want to
> > > > > > > > do
> > > > > > > > > > > > > > something different, both OS-agnostic and
> > > hypervisor-
> > > > > agnostic,
> > > > > > > > > > perhaps
> > > > > > > > > > > > > > we could design a new interface. One that could
> be
> > > > > > > > implementable
> > > > > > > > > > in the
> > > > > > > > > > > > > > Xen hypervisor itself (like IOREQ) and of course
> any
> > > > > other
> > > > > > > > > > hypervisor
> > > > > > > > > > > > > > too.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There is also another problem. IOREQ is probably
> not
> > > be
> > > > > the
> > > > > > > > only
> > > > > > > > > > > > > > interface needed. Have a look at
> > > > > > > > > > > > > > https://marc.info/?l=xen-
> devel&m=162373754705233&w=2.
> > > > > Don't we
> > > > > > > > > > also need
> > > > > > > > > > > > > > an interface for the backend to inject
> interrupts
> > > into
> > > > > the
> > > > > > > > > > frontend? And
> > > > > > > > > > > > > > if the backend requires dynamic memory mappings
> of
> > > > > frontend
> > > > > > > > pages,
> > > > > > > > > > then
> > > > > > > > > > > > > > we would also need an interface to map/unmap
> domU
> > > pages.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My proposal document might help here; All the
> > > interfaces
> > > > > > > > required
> > > > > > > > > > for
> > > > > > > > > > > > > virtio-proxy (or hypervisor-related interfaces)
> are
> > > listed
> > > > > as
> > > > > > > > > > > > > RPC protocols :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > > These interfaces are a lot more problematic than
> > > IOREQ:
> > > > > IOREQ
> > > > > > > > is
> > > > > > > > > > tiny
> > > > > > > > > > > > > > and self-contained. It is easy to add anywhere.
> A
> > > new
> > > > > > > > interface to
> > > > > > > > > > > > > > inject interrupts or map pages is more difficult
> to
> > > > > manage
> > > > > > > > because
> > > > > > > > > > it
> > > > > > > > > > > > > > would require changes scattered across the
> various
> > > > > emulators.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Exactly. I have no confident yet that my approach
> will
> > > > > also
> > > > > > > > apply
> > > > > > > > > > > > > to other hypervisors than Xen.
> > > > > > > > > > > > > Technically, yes, but whether people can accept it
> or
> > > not
> > > > > is a
> > > > > > > > > > different
> > > > > > > > > > > > > matter.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > -Takahiro Akashi
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Oleksandr Tyshchenko