Re: [Xen-devel] libxl/xl memory paging/sharing/ballooning/etc proposal.

["Andres Lagar-Cavilla" <andres@xxxxxxxxxxxxxxxx>]

> I think we've roughly reached consensus on what this could look like.
> I've combined the various aspects of the proposal into the below.

Very awesome. Comments inline below.

>
> In hindsight I should probably have written this as a proposed patch to
> libxl.h and xl.cfg.pod.5 etc but I've written it now so here it is.
>
> There are a few outstanding issues / decisions marked with XXX
>
> Ian J -- I've CC'd you since I propose some new uses of the events
> subsystem (as well as being interested in your overall opinion)
>
> high level libxl interface
> ---------------------------
>
> libxl_domain_set_memory_policy(ctx, domid, const char *actor, const char
> *params):
>
>         Sets the name of the "actor" which will implement memory policy
>         for this domain by writing the name of the actor to
>                 /libxl/X/memory-policy/actor
>         and the params to
>                 /libxl/X/memory-policy/actor-params
>
>         actor-params is defined per-actor. normally be a comma separated
>         key=value list.
>
>         If libxl_domain_set_memory_policy is not called for a domain
>         then the default is considered to be "" i.e. none.
>
>         XXX should the actor config also be part of struct
>         libxl_domain_build_info? (I expect so)
>
> libxl_domain_get_memory_policy(ctx, domid, char **actor, char **params)
>
>         Retrieves the current policy and params (if non NULL etc).
>
> libxl_domain_set_memory_target(ctx, domid, target_memkb, bool relative,
> bool enforce):
>
>         Replaces existing libxl_domain_memory_target() function.
>
>         If memory-policy/actor != "":
>                 updates /libxl/X/memory-policy/<target,enforce>
>         else:
>                 behaves as libxl_domain_memory_target today, e.g. sets
>                 balloon target in xenstore and iff enforcing then sets
>                 maxmem -- probably this is just a call to
>                 libxl_domain_set_balloon_target() (see below)
>
>         target in xenstore is absolute and may be calculated from the
>         existing target and provided target_memkb if relative == true
>
>         enforce in xenstore is the one of the strings "true" or "false".
>         (XXX or 0 1? what do we normally do? does it matter?)
>
> New libxl_event_type DOMAIN_MEMORY_TARGET_CHANGED
>
>         This event is triggered when any of /libxl/X/memory-policy/*
>         change.
>
>         Parameters in libxl_event.u are "char *actor", "char
>         *actor_params", "bool target" and "bool enforce" as read from
>         XS.
>
>         Associated functions libxl_evenable/disable_blah are provided
>         and take an optional "char * actor" to match against. if actor
>         == NULL then events will be generated for any actor name,
>         otherwise only the changes while the actor is the given actor
>         will occur (this includes setting the actor to that value)
>
> On domain build libxl shall behave as if libxl_domain_set_memory_target
> had been called with:
>         target_memkb == libxl_domain_build_info.target_memkb
> (XXX I think that's the right one).
>
> low level libxl interface
> -------------------------
>
> It is expected that toolstacks primary user interface will be via the
> above high level interface. The lowlevel interface is provided in order
> to allow implementation of memory policy actors using libxl and to allow
> toolstacks to optionally provide an interface for expert users to shoot
> themselves in the foot with ( ;-) )
>
> libxl_domain_(set|get)_balloon_target(ctx, domid, target_memkb, relative,
> enforce):
>
>         Read/write /local/domain/X/memory/target to control the in guest
>         balloon driver. This is basically the existing
>         libxl_domain_memory_target with a new name
>
>         As before this is a request to the guest balloon driver to try
>         and use this much actual RAM. This is an existing guest
>         interface which we cannot change.
>
> libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb, relative)
>
>         Read/write whichever XS path controls the xenpaging daemon's
>         target. (Olaf?). I think this path should not be
>         under /local/domain since it should not be guest visible
>         (probably it already isn't).
It is /local/domain/X/memory/target-tot_pages. I agree it should not be
guest visible, and we're still in time to change it.

/libxl/X/memory-policy/paging-target? Is there a non-libxl
non-/local/domain path xenpaging could look into, should it be launched
autonomously? Olaf, do you care about xenpaging in non-libxl environments?

>
>         It is an error to call this for a PV guest (ERROR_??) for a
>         guest which does not have paging enabled
>         (ERROR_PAGING_DISABLED?).
>
> libxl_domain_(enable|disable)_paging(ctx, domid)
>
>         Idempotently starts or stops the xenpaging daemon for a domain.
>

Paging disable takes time. It has to swap back in everything it has paged
out. So it falls under the (really) long-lasting asynchronous operation
framework.

> libxl_domain_(enable|disable)_sharing(ctx, domid)
>
>         Does what it says on the tin.
>
> actor behaviour
> ---------------
>
> An actor should use the libxl event system to monitor for
> DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event for
> which actor != their name.
>
> On receiving such an event the actor should apply its policy and
> configuration in order (try and) cause the domain to use "target" amount
> of actual RAM. This may involve calling the libxl low level interfaces,
> including enabled paging and/or sharing as it desires.
>
> It is an actor specific policy whether it guarantees to meet the target
> (e.g. a balloon only actor would not be able to make that guarantee)
>
> An actor might be either be a single daemon with host wide knowledge or
> a daemon per domain (or anything in between). It is expected in the
> multiple daemon case that all the daemons will either co-ordinate as
> necessary or implement a policy where coordination is not necessary.
>
> Normally the same actor would be used for all domains but we do not rule
> out the possibility for non-homogeneous configurations (but it is up to
> the user to make sure the actors don't have conflicting, potentially
> catastrophic, interactions...)
>
> XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
> actor. This seems like a generally useful event anyhow.

System-wide actors (squeezed) and other actors (xenpaging) already place a
watch on xenstore's @introduceDomain (or whatever it's called). Since this
is not their only use of libxenstore, there is little gain to be made of a
libxl-equivalent event. Referring only to this context.

>
> xl actor configuration
> ----------------------
>
> xl will read its default actor from its configuration file. Since it is
> usually expected to use the same actor for all domains it would normally
> be /etc/xen/xl.conf, but this will be overrideable in a guest config
> file, likewise for the policy params. xl will call
> libxl_domain_set_memory_policy with the configured value while building
> a domain (XXX or use fields in libxl_domain_build_config if we do
> that).
>
> If the configuration key is absent then the default value used is
> "xl" (described below). A user can explicitly supply actor = "" to get
> the default libxl behaviour.
>
> XXX Config key names TBD, "memory_policy" and "memory_policy_args"?
>
> xl high level interface
> -----------------------
>
> xl mem-set <target>
>
>         Pretty much same as today. Calls libxl_domain_set_memory_target
>         with enforcing == 1. (but note that the behaviour of
>         libxl_domain_set_memory_target has changed)
>
> xl mem-policy-set <actor> <param-string>
>
>         Calls libxl_domain_set_memory_policy to set the new actor.
>
> "xl" actor behaviour
> --------------------
>
> Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
> events for the domain which it is managing. When the event is triggered
> (and actor == "xl") then xl will:
>
>         if actor != "xl": return
>
>         call libxl_domain_set_balloon_target(ctx, domid, target,
>         enforcing)
>
>         if domain == pv: return # cannot do more for a PV guest
>
>         XXX if not enforcing: return ??? (not sure about this, we don't
>         actually expose the ability to set enforcing in the xl
>         interface)
>
>         XXX if actor-params:paging!=true return ??? I think we should do
>         this to enable memory_policy_params="paging=true|false" based
>         control of paging.
>
>         delay for a configurable number of seconds (configurable via
>         actor-params:delay=N, default 5s?) XXX does libxl provide a
>         guest accessible timeout event? Would be usefully convenient.

Do you want the guest to try to stop ballooning after the timeout? Just
write /local/domain/X/memory/target to the current value of
xcinfo.nr_pages.

>
>         XXX libxl_domain_paging_enable here or on domain build / policy
>         set? libxl_domain_paging_enable is idempotent so calling it here
>         is always safe?

You can enable the pager on build, unless the actor settings preclude it.
Then all you need to do to get the pager to work is write its xenstore
key.

But generally speaking, the pager might have been disabled ex xl-machina,
so you really want to be able to test_and_set the pager here.

Olaf, does xenpaging have standard daemonize behavior of locking a
/var/foo/pid file?

>
>         call libxl_domain_set_paging_target(ctx, domid, target)
>
> The "xl" actor explicitly does not deal with page sharing. It is assumed
> that page sharing requires system wide knowledge and policy which is out
> of xl's scope and hence a different actor would be needed.
>
> xl low level interface
> ----------------------
>
> All of the following setters functions should error out unless actor ==
> "xl-manual". The getter could return the current value regardless.
>
> xl mem-balloon-(set|get)-target
> xl mem-paging-(set|get)-target
>
>         Set or get the relevant targets by calling the equivalent libxl
>         function.
>
>         Enables or disables paging as necessary, i.e. target == 0 =>
>         disable
>
> NB if you call "xl mem-set" when actor == "xl-manual" then it won't do
> anything since nothing is reacting as that actor.
>
> XXX We could alternatively have xl also implement an "xl-manual" actor
> which immediately sets both ballooning and paging targets to the given
> value. Probably a good idea?

xl-manual makes sense, but setting targets immediately (e.g.
automatically) is an oxymoron :)

>
> XXX Maybe "xl" (the default) should be called "xl-auto" ?

Or xl-manual called "manual"

>
> If you "xl create -e" (or -F) then you won't get a monitoring
> daemon and therefore you won't get any actor implementing the memory
> policy, just like you won't get automatic reboot control. I think this
> is expected and reasonable.

Ok. There actors that conceivably do not depend on libxl events (squeezed,
it can watch on its own memory-policy/target). On one hand they should not
depend on the xl daemon. On the other hand, they would duplicate
functionality that the xl daemon would have to provide anyways, for
monitoring these memory-policy keys.

Thanks Ian,
Andres

>
>
>



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel