Re: Enabling hypervisor agnosticism for VirtIO backends

[Stefan Hajnoczi <stefanha@xxxxxxxxxx>]

On Wed, Aug 25, 2021 at 07:29:45PM +0900, AKASHI Takahiro wrote:
> On Mon, Aug 23, 2021 at 10:58:46AM +0100, Stefan Hajnoczi wrote:
> > On Mon, Aug 23, 2021 at 03:25:00PM +0900, AKASHI Takahiro wrote:
> > > Hi Stefan,
> > > 
> > > On Tue, Aug 17, 2021 at 11:41:01AM +0100, Stefan Hajnoczi wrote:
> > > > On Wed, Aug 04, 2021 at 12:20:01PM -0700, Stefano Stabellini wrote:
> > > > > > Could we consider the kernel internally converting IOREQ messages 
> > > > > > from
> > > > > > the Xen hypervisor to eventfd events? Would this scale with other 
> > > > > > kernel
> > > > > > hypercall interfaces?
> > > > > > 
> > > > > > So any thoughts on what directions are worth experimenting with?
> > > > >  
> > > > > One option we should consider is for each backend to connect to Xen 
> > > > > via
> > > > > the IOREQ interface. We could generalize the IOREQ interface and make 
> > > > > it
> > > > > hypervisor agnostic. The interface is really trivial and easy to add.
> > > > > The only Xen-specific part is the notification mechanism, which is an
> > > > > event channel. If we replaced the event channel with something else 
> > > > > the
> > > > > interface would be generic. See:
> > > > > https://gitlab.com/xen-project/xen/-/blob/staging/xen/include/public/hvm/ioreq.h#L52
> > > > 
> > > > There have been experiments with something kind of similar in KVM
> > > > recently (see struct ioregionfd_cmd):
> > > > https://lore.kernel.org/kvm/dad3d025bcf15ece11d9df0ff685e8ab0a4f2edd.1613828727.git.eafanasova@xxxxxxxxx/
> > > 
> > > Do you know the current status of Elena's work?
> > > It was last February that she posted her latest patch
> > > and it has not been merged upstream yet.
> > 
> > Elena worked on this during her Outreachy internship. At the moment no
> > one is actively working on the patches.
> 
> Does RedHat plan to take over or follow up her work hereafter?
> # I'm simply asking from my curiosity.

At the moment I'm not aware of anyone from Red Hat working on it. If
someone decides they need this KVM API then that could change.

> > > > > There is also another problem. IOREQ is probably not be the only
> > > > > interface needed. Have a look at
> > > > > https://marc.info/?l=xen-devel&m=162373754705233&w=2. Don't we also 
> > > > > need
> > > > > an interface for the backend to inject interrupts into the frontend? 
> > > > > And
> > > > > if the backend requires dynamic memory mappings of frontend pages, 
> > > > > then
> > > > > we would also need an interface to map/unmap domU pages.
> > > > > 
> > > > > These interfaces are a lot more problematic than IOREQ: IOREQ is tiny
> > > > > and self-contained. It is easy to add anywhere. A new interface to
> > > > > inject interrupts or map pages is more difficult to manage because it
> > > > > would require changes scattered across the various emulators.
> > > > 
> > > > Something like ioreq is indeed necessary to implement arbitrary devices,
> > > > but if you are willing to restrict yourself to VIRTIO then other
> > > > interfaces are possible too because the VIRTIO device model is different
> > > > from the general purpose x86 PIO/MMIO that Xen's ioreq seems to support.
> > > 
> > > Can you please elaborate your thoughts a bit more here?
> > > 
> > > It seems to me that trapping MMIOs to configuration space and
> > > forwarding those events to BE (or device emulation) is a quite
> > > straight-forward way to emulate device MMIOs.
> > > Or do you think of something of protocols used in vhost-user?
> > > 
> > > # On the contrary, virtio-ivshmem only requires a driver to explicitly
> > > # forward a "write" request of MMIO accesses to BE. But I don't think
> > > # it's your point. 
> > 
> > See my first reply to this email thread about alternative interfaces for
> > VIRTIO device emulation. The main thing to note was that although the
> > shared memory vring is used by VIRTIO transports today, the device model
> > actually allows transports to implement virtqueues differently (e.g.
> > making it possible to create a VIRTIO over TCP transport without shared
> > memory in the future).
> 
> Do you have any example of such use cases or systems?

This aspect of VIRTIO isn't being exploited today AFAIK. But the
layering to allow other virtqueue implementations is there. For example,
Linux's virtqueue API is independent of struct vring, so existing
drivers generally aren't tied to vrings.

> > It's possible to define a hypercall interface as a new VIRTIO transport
> > that provides higher-level virtqueue operations. Doing this is more work
> > than using vrings though since existing guest driver and device
> > emulation code already supports vrings.
> 
> Personally, I'm open to discuss about your point, but
> 
> > I don't know the requirements of Stratos so I can't say if creating a
> > new hypervisor-independent interface (VIRTIO transport) that doesn't
> > rely on shared memory vrings makes sense. I just wanted to raise the
> > idea in case you find that VIRTIO's vrings don't meet your requirements.
> 
> While I cannot represent the project's view, what the JIRA task
> that is assigned to me describes:
>   Deliverables
>     * Low level library allowing:
>     * management of virtio rings and buffers
>   [and so on]
> So supporting the shared memory-based vring is one of our assumptions.

If shared memory is allowed then vrings are the natural choice. That way
existing virtio code will work with minimal modifications.

Stefan

Attachment: signature.asc
Description: PGP signature