Re: [Xen-devel] libxl/xl memory paging/sharing/ballooning/etc proposal.

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Thu, 2012-03-15 at 14:12 +0000, Andres Lagar-Cavilla wrote:
[...]
> > libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb, relative)
> >
> >         Read/write whichever XS path controls the xenpaging daemon's
> >         target. (Olaf?). I think this path should not be
> >         under /local/domain since it should not be guest visible
> >         (probably it already isn't).
> It is /local/domain/X/memory/target-tot_pages. I agree it should not be
> guest visible, and we're still in time to change it.
> 
> /libxl/X/memory-policy/paging-target? Is there a non-libxl
> non-/local/domain path xenpaging could look into, should it be launched
> autonomously? Olaf, do you care about xenpaging in non-libxl environments?

/xenpaging/N? or /tools/xenpaging/N?

For comparison Roger's hotplug patches seem to be using /hotplug/...

> >         It is an error to call this for a PV guest (ERROR_??) for a
> >         guest which does not have paging enabled
> >         (ERROR_PAGING_DISABLED?).
> >
> > libxl_domain_(enable|disable)_paging(ctx, domid)
> >
> >         Idempotently starts or stops the xenpaging daemon for a domain.
> >
> 
> Paging disable takes time. It has to swap back in everything it has paged
> out. So it falls under the (really) long-lasting asynchronous operation
> framework.

Good point, thanks.

> 
> > libxl_domain_(enable|disable)_sharing(ctx, domid)
> >
> >         Does what it says on the tin.

Also long running?

> >
> > actor behaviour
> > ---------------
> >
> > An actor should use the libxl event system to monitor for
> > DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event for
> > which actor != their name.
> >
> > On receiving such an event the actor should apply its policy and
> > configuration in order (try and) cause the domain to use "target" amount
> > of actual RAM. This may involve calling the libxl low level interfaces,
> > including enabled paging and/or sharing as it desires.
> >
> > It is an actor specific policy whether it guarantees to meet the target
> > (e.g. a balloon only actor would not be able to make that guarantee)
> >
> > An actor might be either be a single daemon with host wide knowledge or
> > a daemon per domain (or anything in between). It is expected in the
> > multiple daemon case that all the daemons will either co-ordinate as
> > necessary or implement a policy where coordination is not necessary.
> >
> > Normally the same actor would be used for all domains but we do not rule
> > out the possibility for non-homogeneous configurations (but it is up to
> > the user to make sure the actors don't have conflicting, potentially
> > catastrophic, interactions...)
> >
> > XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
> > actor. This seems like a generally useful event anyhow.
> 
> System-wide actors (squeezed) and other actors (xenpaging) already place a
> watch on xenstore's @introduceDomain (or whatever it's called). Since this
> is not their only use of libxenstore, there is little gain to be made of a
> libxl-equivalent event. Referring only to this context.

What other use of xenstore do they make? Depending what those are it's
possible that it may fall under the remit of libxl to provide such
interfaces.

It should be possible for an actor to be only implemented using libxl,
which I think requires us to provide such an event. This doesn't
preclude some actors from using facilities from other places too though.

[...]
> > "xl" actor behaviour
> > --------------------
> >
> > Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
> > events for the domain which it is managing. When the event is triggered
> > (and actor == "xl") then xl will:
> >
> >         if actor != "xl": return
> >
> >         call libxl_domain_set_balloon_target(ctx, domid, target,
> >         enforcing)
> >
> >         if domain == pv: return # cannot do more for a PV guest
> >
> >         XXX if not enforcing: return ??? (not sure about this, we don't
> >         actually expose the ability to set enforcing in the xl
> >         interface)
> >
> >         XXX if actor-params:paging!=true return ??? I think we should do
> >         this to enable memory_policy_params="paging=true|false" based
> >         control of paging.
> >
> >         delay for a configurable number of seconds (configurable via
> >         actor-params:delay=N, default 5s?) XXX does libxl provide a
> >         guest accessible timeout event? Would be usefully convenient.
> 
> Do you want the guest to try to stop ballooning after the timeout? Just
> write /local/domain/X/memory/target to the current value of
> xcinfo.nr_pages.

Could do, I don't mind.

Leaving it alone means that a guest which becomes co-operative "too
late" will eventually reduce the amount of paging it is subjected to to
0. Doing as you suggest stops the guest waking up every $PERIOD to try
ballooning a bit more.

Could be an actor-param, although it would be better if xl could just Do
The Right Thing.

> >         XXX libxl_domain_paging_enable here or on domain build / policy
> >         set? libxl_domain_paging_enable is idempotent so calling it here
> >         is always safe?
> 
> You can enable the pager on build, unless the actor settings preclude it.
> Then all you need to do to get the pager to work is write its xenstore
> key.

My thinking was that if you boot "unpaged" you might want it to be
enabled when you call mem-set. Likewise if you mem-set such that no
paging is required you might like the daemon to go away. I think the
former is more important than the latter. We don't really want to run
xenpaging for every HVM domain even when it isn't used, whereas it is
fine if it persists after it has been used.

> 
> But generally speaking, the pager might have been disabled ex xl-machina,
> so you really want to be able to test_and_set the pager here.
> 
> Olaf, does xenpaging have standard daemonize behavior of locking a
> /var/foo/pid file?

If it is running this should be indicated in the same XS path as we
choose above?

> >         call libxl_domain_set_paging_target(ctx, domid, target)
> >
> > The "xl" actor explicitly does not deal with page sharing. It is assumed
> > that page sharing requires system wide knowledge and policy which is out
> > of xl's scope and hence a different actor would be needed.
> >
> > xl low level interface
> > ----------------------
> >
> > All of the following setters functions should error out unless actor ==
> > "xl-manual". The getter could return the current value regardless.
> >
> > xl mem-balloon-(set|get)-target
> > xl mem-paging-(set|get)-target
> >
> >         Set or get the relevant targets by calling the equivalent libxl
> >         function.
> >
> >         Enables or disables paging as necessary, i.e. target == 0 =>
> >         disable
> >
> > NB if you call "xl mem-set" when actor == "xl-manual" then it won't do
> > anything since nothing is reacting as that actor.
> >
> > XXX We could alternatively have xl also implement an "xl-manual" actor
> > which immediately sets both ballooning and paging targets to the given
> > value. Probably a good idea?
> 
> xl-manual makes sense, but setting targets immediately (e.g.
> automatically) is an oxymoron :)
> 
> >
> > XXX Maybe "xl" (the default) should be called "xl-auto" ?
> 
> Or xl-manual called "manual"

I think namespacing the actors makes sense myself, there's no harm?

Perhaps <actor>-<policy> would be a good guideline for naming the thing
we've been calling the actor?

> > If you "xl create -e" (or -F) then you won't get a monitoring
> > daemon and therefore you won't get any actor implementing the memory
> > policy, just like you won't get automatic reboot control. I think this
> > is expected and reasonable.
> 
> Ok. There actors that conceivably do not depend on libxl events (squeezed,
> it can watch on its own memory-policy/target). On one hand they should not
> depend on the xl daemon.

Squeezed should eventually be implemented in terms of the mechanisms
discussed here IMHO. This would remove the duplication of mechanism,
leaving only the policy. Of course if the author of an actor wants to
use other mechanisms then fine.

Also bear in mind that the toolstack might not be xl, i.e. xapi should
eventually be ported to use the highlevel libxl interface and would
likely choose to set "squeezed" as its default actor.

>  On the other hand, they would duplicate
> functionality that the xl daemon would have to provide anyways, for
> monitoring these memory-policy keys.

If actor="squeezed" then, even if it is the toolstack, xl will not be
providing any mechanism, since it's own actor will not do anything when
actor=="squeezed".

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel