Re: Virtio Xen (WAS: Re: [Linux] [ARM] Granting memory obtained from the DMA API)

[Julien Grall <julien@xxxxxxx>]

On 29/08/2020 11:38, Simon Leiner wrote:
> Hi Julien,
Hi Simon,

Apologies for the late answer.

> On 25.08.20 15:02, Julien Grall wrote:
> > May I ask why did you create a new transport rather than using the
> > existing one?
> We wanted a mechanism for dynamically creating virtio devices at
> runtime. I looked at virtio-mmio briefly and it seemed to me that a lot
> of things would have to be known in advance (how many devices are
> there? How much memory do they need? Where does the memory range for
> virtio-mmio start on the device domain?). So after reading a bit about
> Xen and how the classic split drivers work, I figured building a split
> driver for virtio was the way to go. The basic principle is really
> simple:
>
>   - Using grants to share memory for the virtqueues
>   - Using event channels as a queue notification mechanism
>   - All state handling and other information exchange (like number of
>     queues, grant refs, event channel numbers etc.) is done through the
>     Xenbus.
>
> On the Linux side, this is implemented as a kernel module. No patches
> to the kernel itself (apart from the ones I sent in earlier) or to Xen
> itself are required.
>
> > So far, there is an RFC to implement virtio-mmio for Xen on Arm
> I did not see that before. Also, I'm not familiar with the ioreq
> mechanism. But from skimming the patch, it seems like it achieves the
> same thing (dynamic creation of virtio devices at runtime). Is that
> right?
I am not aware of any mechanism with virtio-mmio to notify a new device 
after the OS booted. But virtio-pci should allow you to do as as this is 
just a PCI device.
However, you will still need to size the PCI hostbridge I/O region 
correctly when the domain creation. Not sure if this would be an issue 
in your use case.
> > But the idea of a Xen specific transport is discussed on the mailing
> > list time to time. It would be more secure than existing transport,
> > but I am worried about the adoption of the transport.
> What are the security issues with the existing transport mechanisms?
In the Xen PV model, a backend cannot access the frontend memory unless 
the latter did explictly grant.
In the default virtio-{mmio, pci} model, a backend must have full access 
to the frontend memory. This can be an issue if you don't trust your 
backend or it can get compromised.
> I'm quite new to the Xen community, so I have no idea about adoption
> rates.
>
> > A new transport requires to modify all the OSes so they can work on
> > Xen.
> Just to be clear: They would need to be modified in order to support
> that mechanism, but it changes nothing about the interface between
> hypervisor and guest.
Right, this is the first bit I am more concerned about. IMHO, the main 
goal of virtio is to allow moving from one hypervisor to another without 
(or possibly limited) changing the guest OS code.
Adding a new transport open source OS for a new transport is fairly 
easy, but it may be harder to justify for proprietary OS if it only 
benefits Xen.
That said, if we can convince other hypervisor vendors to adopt it then 
most of my concern are moot :).
> However, supporting the same use case with an already existing
> transport mechanism is more elegant than building another transport
> mechanism specifically for that case IMO. The only technical difference
> between my implementation and the virtio-mmio approach in actually
> running the virtio device is that I'm using event channels for queue
> notification while virtio-mmio uses some bytes in memory for that. I do
> not have any measurements indicating whether or not this makes a
> difference.
The virtio-mmio notification is potentially going to be expensive on Xen 
because the guest because a vCPU will be blocked until the I/O has been 
handled by the IOREQ server.
The notification would look like:
    1) Frontend write in notification address
    2) Trap in Xen
    3) Pause the vCPU and forward the I/O to the IOREQ server (e.g. 
your virtio backend)
    4) Schedule the domain where the IOREQ server resides
    5) Wait for the I/O completion
    6) Unpause the vCPU
    7) Return to guest

We may be able to optimize as there is no need to wait for the I/O to 
complete when we notify.
Cheers,

--
Julien Grall