[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] On distro packaging of stub domains (Re: Notes from Xen BoF at Debconf15)



Hi,

Wei Liu hinted that I should "chime in and / or provide corrections" (his words). I'll attempt to do exactly that by not really replying to anything specific. For the record, when I say "we" in this mail, I mean "people who have contributed to the rump kernel project" (as also indicated by the email-hat).

First of all, there's a difference between a rump kernel (driver bundle built out of unmodified kernel components) and any unikernel you construct out of rump kernels ... sort of like how there's a difference between Linux and GNU/Linux.

For unikernels, the rump kernel project provides Rumprun, which can provide you with a near-full POSIX'y interface. Rumprun also provides toolchain wrappers so that you can compile existing programs as Rumprun unikernels. Rumprun also recently regrew the ability to run without the POSIX'y bits; some people found it important to be able to make a tradeoff between running POSIX'y applications and more compact "kernel plane" unikernels such as routers and firewalls. But, for brevity and simplicity, I'll assume the POSIX'y mode for the rest of this email, since that's what the QEMU stubdom will no doubt use.

If the above didn't explain the grand scheme of things clearly, have a look at http://wiki.rumpkernel.org/Repo and especially the picture. If things are still not clear after that, please point out matters of confusion and I will try to improve the explanations.

Also for simplicity, I'll be talking about rump kernels constructed from the NetBSD kernel, and the userspace environment of Rumprun being NetBSD-derived. Conceptually, there's nothing stopping someone from plugging a GNU layer on top of NetBSD-derived rump kernels (a bit like Debian kXBSD?) or constructing rump kernels out of Linux. But for now, let's talk about the only working implementation.

As far as I know, the API/ABI of the application environment provided by Rumprun is the same as the one provided by standard NetBSD. Granted, I didn't perform the necessary experiments to verify that, so take the following with a few pinches of salt. In theory, you could take application objects built for NetBSD and link them against Rumprun libs. However, since a) nobody (else) ships applications as relocatable static objects b) Rumprun does not support shared libraries, I don't know how helpful the fact of ABI compatibility is. IMO, adding shared library support would be a backwards way to go: increasing runtime processing and memory requirements to solve a build problem sounds plain weird. So, I don't think you can leverage anything existing.

We do have most of the Rumprun cross-toolchain figured out at this point. First, we don't ship any backend toolchain(s), but rather bolt wrappers and specs on top of any toolchain (*) you provide. That way we don't have to figure out where to get a toolchain which produces binary for every target that everyone might want. Also, it makes bootstrapping Rumprun convenient, since you just say "hey give me the components and application wrappers for CC=foocc" and off you go.

*) as long as it's gcc-derived, for now (IIRC gcc 4.8 - 5.1 are known to work well, older than that at least C++ won't work). clang doesn't support specs files at least AFAIK, so someone would have to figure out how to move the contents of the specs into the wrappers, or whatever equivalent clang uses. (patches welcome ;)

The produced wrappers look exactly like a normal cross-toolchain. The tuple is the same as what NetBSD uses, except with rumprun added in the middle, so e.g. x86_64-rumprun-netbsd or arm-rumprun-netbsdelf-eabihf. That naming scheme means that most GNU-using software compiles nicely for Rumprun just by running configure as ./configure --host=x86_64-rumprun-netbsd followed by "make". Sometimes you additionally need things like --disable-shared, but all in all everything works pretty well. See http://repo.rumpkernel.org/rumprun-packages for a bunch of "case studies", not limited to just GNU autotools.

After "make", before launch we have an additional step called "bake", which links the specific kernel components onto the binary. So for example, you can make the compiled binary run on Xen or KVM depending on which kernel components you bake onto it. As a crude analogy, it's like scp'ing a binary to a Xen or KVM or bare metal system, but since the Rumprun unikernel is missing exec, we use the linker to perform "system selection".

So for shipping, one option is to ship the binary after "make", but then you also need to ship the toolchain. The other option is to ship the baked binary, but then you lose some of your possibilities on how to target the binary. I'm not sure either option is right for all cases.

We're still trying to figure out the exact form and figure of bake+launch. In the original implementation we assumed that at launch-time we could cheaply control all of the details of the backend (a la xl.conf). That assumption proved to be bad not only for example for embedded systems (which we should've foreseen), but also for cases like Amazon EC2 (where creating something launchable and launching something are seriously separate steps). I'm not going to go into details in this thread, but just saying that we still have a few things to figure out in the full source-to-execution chain. If someone wants to ship something, please please check with the rump kernel community first so that we know that you want to depend on some syntax/semantics not changing anymore.

I don't really have good solutions for the packaging problem. Building a "full distro" around rump kernels certainly sounds interesting, and we're sort of trying to experiment with that with rumprun-packages. However, for packages sooner or later we need to assimilate a real packaging system which properly manages dependencies, licenses, vulnerabilities, etc. I'm unsure we'd want to assimilate every existing packaging system -- Rumprun works the same on every platform, after all. Hard to say for now, I hadn't actually considered the case where a distro might want to natively ship binary Rumprun images, as opposed to just the toolchain and components.

If I were you folks, I'd start by getting qemu out of the door, and worry about generalized solutions when someone wants to ship the second unikernel (or third or any small N>1). If you can't sell to distros something that solves a problem, it's unlikely you'll be able to sell a framework for solving the same one problem (though, granted, it might be easier to sell a framework to computer folk -- "nevermind the solution, here's abstraction!").

If you haven't tried rumprun-packages, I recommend to pick something from there and see how it works. Doing so might give some more perspective on how easy or difficult it would be to package QEMU.

Whoops, ended up being a bit longer than what I hoped for, but with any luck at least some new information was communicated.

  - antti

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.