Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

On 18 Mar 2015, at 11:20, Martin Lucina <martin@xxxxxxxxxx> wrote:
>> 
>> A bit background information. A stubdom is a service domain.  With QEMU
>> stubdom we are able to run QEMU device emulation code in a separate
>> domain so that bugs in QEMU don't affect Dom0 (the controlling domain).
>> Xen currently has a QEMU stubdom, but it's based on our fork of ancient
>> QEMU (plus some other libraries and mini-os). Eventually we would like
>> to use upstream QEMU in stubdom.
>> 
>> I've now successfully built QEMU upstream with rump kernel. However to
>> make it fully functional as a stubdom, there are some missing pieces to
>> be added in.
>> 
>> 1. The ability to access QMP socket (a unix socket) from Dom0. That
>>   will be used to issue command to QEMU.
>> 2. The ability to access files in Dom0. That will be used to write to /
>>   read from QEMU state file.
> 
> As I understand from Stefano's and Anthony's replies in this thread, both
> of the above can be implemented using an AF_UNIX or AF_INET socket on the
> QEMU end. Such an implementation would not require anything special done in
> QEMU, just telling it which socket to use using existing mechanisms.
> 
> So, let's step back a bit: What we need is a trusted communication channel
> from a Rump Kernel domU to dom0, using existing socket or socket-like[*]
> APIs at both the domU and dom0 ends.
> 
> This fits in with a couple of things I hope to make time to work on in the
> next couple of months:
> 
> 1. Introspection of Rump Kernel domUs for ops purposes, i.e. get some
>    basic "ps", "top", "vmstat"-like information about what the domU is
>    doing from the dom0.
> 
> 2. Connecting up multiple Rump Kernel domUs and/or Mirage domUs. The
>    general idea here is that you can have e.g. a Mirage domU running a
>    HTTP+TLS frontend, communicating with a Rump Kernel domU running PHP +
>    FastCGI.
> 
>    The Mirage folks are already doing something similar in their 
>    Jitsu work, using a protocol called Conduit which runs over vchan.

Yeah, this is currently requiring a couple of things:

- kicking the tires with Vchan and its associated machinery, which has
  taken some time.  Dave Scott has built a complementary system for
  the xentropyd which simply sets up a console ring instead of vchan.
  This has the drawback of being a single fixed page, but far simpler.

- A XenStore protocol for setting up stream connections.  This could
  indeed quite easily turn into a AF_VCHAN that could be transparently
  used by rump/Mirage/HaLVM and normal domUs for VM<->VM comms.

> Now, both of the above require exactly the same underlying mechanism.
> 
> Point 2. will further require implementing support in the Rump Kernel,
> either for a shim which would proxy AF_UNIX / AF_INET transparently using
> vchan, or possibly later implementing a separate socket family (AF_VCHAN /
> AF_HYPER?). Once that is done you should be able to just drop it in to
> QEMU on Rump.

I'm a little wary of point 2) asking for filesystem access to dom0.  What
exactly is the qemu state API?  Does it need arbitrary file access, or is
there a slightly higher level set of operations that could be marshalled
along the socket?  In fact, why doesn't qemu privilege separate and use
a QMP socket for its host filesystem operations as well?

> 
> [*] Aside: What I mean by socket-like is that the implementation does not
> need to be in the dom0 kernel, it can just be a user-space library. For
> example, see the nanomsg or ZeroMQ APIs, which I have worked on extensively
> in the past.
> 
>> 3. The building process requires mini-os headers. That will be used
>>   to build libxc (the controlling library).
> 
> As Antti already suggested, if you can use POSIX interfaces rather than
> mini-os ones in QEMU, then that would be a better approach.
> 
>> One of my lessons learned from the existing stubdom stuffs is that I
>> should work with upstream and produce maintainable code. So before I do
>> anything for real I'd better consult the community. My gut feeling is
>> that the first two requirements are not really Xen specific. Let me know
>> what you guys plan and think.
> 
> Thanks for getting in touch. I think this is an important discussion!

Very much so -- the time is definitely right to establish some unikernel
interop standards.  I'm also looking forward to better rump<->MirageOS
comms in particular (shiny new protocol stacks working alongside existing
applications in separate VM containers).

-anil

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel