Re: [Xen-devel] qemu-upstream stubdom - status?

[Eric Shelton <eshelton@xxxxxxxxx>]

On Tue, Apr 8, 2014 at 7:26 AM, Anthony PERARD
<anthony.perard@xxxxxxxxxx> wrote:
> On Tue, Apr 08, 2014 at 11:49:43AM +0100, George Dunlap wrote:
>> On Mon, Apr 7, 2014 at 9:50 PM, Eric Shelton <eshelton@xxxxxxxxx> wrote:
>> > Where does this feature stand currently?  It pops up under a couple of 
>> > names
>> > and proposed implementations (e.g., "qemu-xen stubdomains", "qemu-upstream
>> > stubdom, Linux", and "qemu-upstream stubdom, BSD libc"), so it seems likely
>> > I missed something.
>> >
>> > If I understand this email from Sept 2013
>> > (http://lists.xen.org/archives/html/xen-devel/2013-09/msg02881.html),
>> > qemu-xen stubdomains might have made it into 4.4 if the schedule had been
>> > slipped a little, suggesting some active effort.  On the other hand, the
>> > same email designated the efforts attached to Anthony and Ian as 
>> > "prognosis:
>> > ?".  Anthony released a patchset/branch about a year ago, but that got
>> > sidelined in favor of getting Xen on ARM.  However, I did not identify any
>> > more recent patches.
>> >
>> > It remains appealing to make use of the more recent developments in
>> > qemu-upstream, but with the isolation provided by a stubdomain.  The many
>> > improvements made in teasing driver domains out of dom0 strengthen this
>> > appeal (for example, using OpenSolaris as a ZFS-based storage domain).
>> >
>> > What are the most recent patches or repository for this feature?
>>
>> I think stubdomains for qemu-xen should be a blocker for 4.5.
>>
>> The only thing that's missing for stubdoms, AFAIK, is an integrated
>> way to deploy the stubdom VM in which qemu-xen can run.
>> qemu-traditional uses minios with a version of newlib (IIRC) ported to
>> it.  The problem at the moment is that qemu-xen requires much more
>> from its libc than newlib provides.  There are several potential
>> solutions that have been investigated:
>>
>> 1. Use Linux with glibc.
> [...]
>
>> #1 involves mainly trying to cut down the Linux kernel and the guest
>> FS as small as possible.  Anthony spent some time on #1 in the 4.3 and
>> 4.4 development cycles, and I think got the image down to 32MB.
>> That's still fairly large, however; and I think he got stuck trying to
>> integrate an image builder into the Xen build system.
> [...]
>
>> If you're looking for a solution to use right now, I'm sure Anthony
>> can point you to wherever he left off.
>
> I have this branch that is the most recent work I have:
> git://xenbits.xen.org/people/aperard/xen-unstable.git
> branch: stubdom
>
> `make -C stubdom-linux` should make you an image.
>
> Unfortunatly, there is no video output that can be seen, so a guest is
> only accessible through the serial console or the network. And there are
> probably many other issues.
>
> Hope that help,
>
> --
> Anthony PERARD

With the impending rollout of another Xen release lacking qemu-xen
stubdom, I would like to campaign for a couple of things: (1) making
qemu-xen stubdom a blocker for Xen 4.6; and (2) using a Linux-based
stubdom, at least for the time being.

(1) Making qemu-xen a blocker for 4.6

This security issues presented by allowing qemu to run unrestricted
within dom0 have been appreciated for a long time (comments about
qemu-dm in XSA-109 illustrate this).  Just last month, a few
additional qemu escalation vulnerabilities were demonstrated.  Given
the size, complexity, and pace of development of qemu, we can
reasonably assume there will always be some escalation vulnerability.

Xen has generally taken known, and often far more obscure, security
issues very seriously.  I am puzzled that Xen continues to offer a
non-stubdom qemu.  The current qemu stubdom solution almost amounts to
a mitigation, rather than a solution, of a well-known issue.  The
options offered today are: (a) have access to all of the features and
bugfixes offered by upstream qemu (which have been significant thanks
to KVM's use of qemu, and are appreciable on and sometimes necessary
for some guest operating systems), at the cost of qemu running
unrestricted within dom0; or (b) have qemu contained in a separate
domain, but with what is essentially only a "good enough" six year old
version of qemu (ver 0.10.2).

Additionally, once qemu-xen stubdom is realized, we can move away from
and stop maintaining the forked qemu-traditional codebase.  Finally,
all of the Xen-specific code could be mainstreamed into upstream qemu.

If all that can managed is to release qemu-xen stub domains at the
level of a tech preview, it still represents a significant improvement
on both of the axes of functionality and security.

(2) Using Linux to implement qemu-xen stubdom

Efforts over the last 3 years at realizing qemu-xen stub domains seem
to illustrate the "perfect is the enemy of the good" phenomenon.  I
gather that rump kernel is the favored direction, and ultimately would
be more memory efficient than using Linux, but a lot of unresolved
technical issues lie down that path that are already solved by using
Linux, and resolving them has, and likely will continue to, stymie
getting something effective and reasonable out the door.

It is understandable that the Xen team has prioritized other items
over qemu-xen stubdom.  However, rather than allowing a lower priority
item to be left undone, that should inform us that an "ideal," but
more development intensive, path is not called for.  Instead, we
should just adopt the more easily implemented solution and move on.

Some reasons for picking up where Anthony left off with using Linux:
(a) Linux + qemu is a mature and well-tested codebase.  Although there
are some other environments under which qemu is built and run, far and
away the most common platform is Linux.
(b) Linux kernel + Xen is a mature and well-tested codebase.  Although
there was some disappointment that external patches had to be applied
to the Linux kernel, they were pretty minor.
(c) A lot things need to be done for rump kernel that already just
work with Linux.
(d) Developers are more familiar with Linux.  At one point along the
way, there was expressed interest in moving away from the obscure
mini-os.  With rump kernel, mini-os remains in the picture AND we are
adding NetBSD into the mix.  There are very few developers with the
skills needed to make rump kernel happen or contribute to it.
(e) The changes that need to be made to upstream qemu to run in a stub
domain are a fixed item - they are somewhat independent of the
underlying OS/execution environment.  If Xen starts with a Linux-based
stubdom, the transition can still be made to rump kernel.
(f) The memory overhead issue is overblown.  As time goes on, systems
have more memory and memory gets cheaper.  How hard do we work to
reduce the 32 MB used for Linux when we're allocating 2+ GB of memory
to an HVM domain anyway, and how much of a reduction would we realize
anyway?

When Anthony released his patches, there was some disappointment
expressed with the build process (which kernel version was being used
and reliance on busybox, for example).  None of the raised issues
strike me as significant problems - just decisions that have to be
made.  Anyways, I find it incredible to believe that a MiniOS + NetBSD
rump kernel + libc + whatever else solution is going to result in a
cleaner build process.

Given limited developer resources, sometimes tradeoffs have to be
made.  I propose that Linux-based stub domains are the right tradeoff
for incorporation into Xen 4.6.  Rump kernels may indeed be the
future, but they do not seem to be the immediate solution for qemu
stubdoms that we could use.

Thank you,
Eric

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel