[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] XSA-180 follow-up: repurpose xenconsoled for logging



On Mon, Jun 06, 2016 at 02:03:45PM +0100, Andrew Cooper wrote:
> On 06/06/16 11:12, George Dunlap wrote:
> > On 03/06/16 18:38, Andrew Cooper wrote:
> >> On 01/06/16 15:00, Wei Liu wrote:
> >>> Hi all
> >>>
> >>> During the discussion of XSA-180 Ian came up with the idea that we
> >>> repurpose xenconsoled to handle logging. I've done some research (and
> >>> some coding as well!).
> >>>
> >>> In my reply to George the other day:
> >>>
> >>>> I just read the code of virtlogd and xenconsoled.
> >>>>
> >>>> I think xenconsoled is missing at least things.
> >>>>
> >>>> From a higher level:
> >>>>
> >>>> 1. Abstraction of rotating file.
> >>>> 2. Abstraction of client.
> >>>> 3. IPC interface to libxl -- presumably we need to create a socket.
> >>>>
> >>> I've done #1 and port existing code to use that -- would be useful in
> >>> general.
> >>>
> >>> #2 is not too hard because xenconsoled already has concept of a domain.
> >>> I suspect some refactoring will be fine.
> >>>
> >>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
> >>> pretty sure we *don't* want that for xenconsoled. So I spent some time
> >>> this morning and write up a draft for a xenstore based protocol. See
> >>> below.
> >>>
> >>> Also there is an implication here: we put xenconsoled in a really
> >>> critical path. If for some reason it doesn't work all guests are
> >>> blocked. Do we really want to do this?
> >> Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
> >> now done.
> >>
> >> Architecturally, this is a bad idea, and -1 from me.
> >>
> >> First of all, you are making the assumption that xenconsoled and all
> >> qemus are in the same domain.  This is not necessarily the case,
> >> although I suppose that `xl console` does depend on xl and xenconsoled
> >> being in the same domain.
> >>
> >> At the moment, if xenconsoled crashes, the worst that happens is that
> >> you cant interact with guest consoles, and the guests will notice that
> >> their console rings aren't being drained.  This is a safe, albeit
> >> suboptimal, situation which allows guests to continue running unimpeeded.
> >>
> >> If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
> >> block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
> >> As you identify, there is a security implication here, where a guest
> >> which can crash xenconsoled can also DoS all other HVM domains in the
> >> system.
> >>
> >> From the scalability side, XenServer supports running 1000 windows VMs
> >> on a single host (subject to sufficient RAM), and xenstored is a big
> >> monlithic single point of failure.  We would like to avoid adding a second.
> > qemu stubdoms is a red herring; that's a separate codepath which will be
> > modified appropriately for its use case.
> 
> Most certainly not (and I didn't specify qemu stubdoms).  What happens
> if someone wishes to run xenconsoled in a separate non-stub domain? 
> What happens if someone has separate non-stub domains for running qemus?
> 
> All of these are currently viable options with XSM.
> 

The scalability issue is true. TBH I'm also a bit skeptical to the
scalability issue. But in the end I think it wouldn't be any different
if you want to use syslog or any other daemon. The issue is going to be
the there if you opt to use any daemon to handle log.

There is a difference between requiring QEMU and the logging daemon to
live in the same domain v.s.  requiring the toolstack that sets up
logging for QEMU knows how to do it. That is, the whatever toolstack to
spawn QEMU in the other domain will know how to handle logging anyway.
Running QEMU and xenconsoled in the same domain won't be a hard
requirement. It is fair to expect xenconsoled and the toolstack live in
the same domain (TOOLSTACK_DOMAIN), so it would be fair to expect
xenconsoled to be available when libxl tries to start QEMU.

I wouldn't expect xenconsoled to crash easily.

> > The potential for qemu to
> > block if the daemon dies, as well as scalability concerns, are good
> > points which should be considered; and we certainly need to allow other
> > ways of dealing with logging.
> >
> > But...
> >
> >> If someone builds Xen from source and finds later that their disks are
> >> full up with logs, then tough.
> > ...I cannot express how wrong-headed I think this statement is; and...
> 
> That is merely your opinion.  You know mine, and I stand by it.
> 

I don't think these two views are fundamentally in conflict with each
other. To provide a system level solution, we need to provide something
to be integrated with the system. There is none so far.

> >
> >> If someone installs Xen from a distro,
> >> they should reasonably expect logging, and log management, to be
> >> consistent with other packages from the same distro.
> > ...the degree to which this is true depends on the degree to which
> > distributions are actually equipped to deal with these sorts of things.
> 
> A distro which isn't equipped to deal with these things is of no
> practical use in the real world.
> 
> Any distro we care about supporting is equipped to deal with logs.
> 
> >
> > I agree that if logging using existing logging systems were workable,
> > that would be a better solution.  Wei did actually spend a decent chunk
> > of time looking at other options, including logrotate and journald
> > before settling on using xenconsoled.
> >
> > A lot of this brainstorming and discussion happened off-list because
> > XSA-180 was still embargoed, so I can understand why it looks like this
> > came out of nowhere.  It would probably be good for Wei to report here
> > what he found and why he decided to propose this solution instead.
> 
> Please do.  Until there is an understanding of why the standard
> mechanisms are not suitable, it is premature and naive to re-invent a wheel.
> 

With syslog and logrotate you will still end up filling up your disk.
Logrotate can't actively rotate log files.

You can't just tap syslog to QEMU at the moment unless you use the
script I sent to XSA-180 security@ discussion. That's still a hacked up
solution.

I actually don't mind having syslog deal with those, but we need to
provide some not-so-hacked-up way for doing it.

> >
> > FWIW, the libvirt project has exactly the same problem, and they did the
> > analog of what Wei is proposing -- they added a new daemon, virtlogd, to
> > handle all the console and debug log rotation in a fashion resistant to
> > DoSing.  Without reading their discussion, it's reasonable to assume
> > that using system logging was at least considered using system-level
> > logging before deciding to write their own code.
> 
> Just because a project followed this path, doesn't mean it is a sensible
> thing to do.
> 
> >
> > We already have a daemon to do logging of consoles; it just doesn't have
> > any of the logrotate features that are needed to make it robust against
> > DoS.  There's no sense in having log rotation code in two places, so
> > upgrading xenconsoled to do what virtlogd is doing makes more sense than
> > say, either writing our own, or stealing virtlogd.
> 
> We have a daemon for connecting local pty's to guest consoles, which has
> a disabled-by-default option to log consoles to files.
> 
> >
> >> I think the best solution here is to manage expectations, not to follow
> >> the kneejerk reaction and hack up a solution.
> > Hopefully you can see now that this is not a knee-jerk solution. :-)
> > Thank you for making sure other options is properly discussed in public.
> >
> > Tying off a few random technical threads:
> >
> > What does xapi do at the moment?
> 
> Plain syslog and logrotate.  No frills.

How does XAPI redirect QEMU's stderr to syslog?

If it isn't too complex I would like to add that to libxl regardless of
whether we want to use xenconsoled or not.

> 
> > Re the interface, it seems like having the option to log to {nothing,
> > file, pipe, xenconsoled} should cover the reasonble usecases: libvirt
> > for instance could take the pipe and pipe it to virtlogd (and xapi could
> > do something similar if it wanted).
> 
> My main concern is that this is re-inventing a wheel which noone will use.
> 

It's going to be used by a default installation of open source Xen.

But, I would like to devise a mechanism that system administrator can
opt out.

> >
> > Regarding qemu grinding to a halt if xenconsoled dies: That may be
> > something worth considering; but at least on Linux you can set pipes as
> > O_NONBLOCK and you'll get EAGAIN when writing to a full pipe instead of
> > blocking.  We'd have to make sure that the error didn't cause any other
> > failures (ignoring it and letting the write drop is probably the best
> > thing), but that shouldn't be terribly difficult to do.
> 
> stderr really shouldn't be in a position where it gets -EAGAINs in a
> common case.
> 

I agree.

Wei.

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.