Hello all,
I'm a Masters student at UBC and I've been doing some work on XenStore
for the last few months. In doing so, I have come across some
idiosyncrasies and annoyances with it.
I'm planning on doing a re-write of it to see if some of these can be
resolved and would like input from the community.
First of all, I will list some issues I have come across with XenStore:
Transactions:
- Currently the entire TDB database (more on TDB later) is copied on
each transaction, which is really slow and unneeded
- Interleaving transactions cause EAGAIN to be issued regardless of
whether the transactions actually conflict or not
- There is no support for nested transactions
Watches:
- Apparently when a domain disconnects and reconnects, its watches
aren't deleted from XenStore, but nor can it access them again upon
reconnect
- Watches cannot be set on non-existent directories. While this makes
sense, it causes performance problems with some devices that require a
watch placed on a path that hasn't yet been created. The solution is to
place a watch on / and check each event for the creation of the desired
path, which can cause excessive amounts of unneeded watches
Code:
- Things that start with t are bad
- TDB is a bad choice for a backend
- talloc is a pain to use
- Policy and mechanism are completely tangled
- This should be separated out, then different policy modules can be
implemented and trivially enabled (e.g. legacy, Chinese Wall, protocol
enforcement)
- Modularity -- related to the above comment, but with greater scope
- The backend should be pluggable (e.g. TDB, in-memory-only store,
flat file, sqlite, anything else you want)
My intention is to rewrite XenStore using OCaml in order to provide
stronger assertions about the code and the type system. An interface
will be provided so that modules can be written in C and plugged into
the core XenStore code.
I've also noticed some aspects of the XenStore protocol which I think
might be good to look at.
I notice that XenStore has RELEASE and RESUME commands, but according to
the current XenStore documentation (which I have also modified/cleaned
up to reflect an implementation independent view of the XenStore API and
will follow later in this message) there are no current users of RESUME
in the xen-unstable source. In addition, some other work done here on a
project called Remus has found that XenStore's RELEASE/RESUME
functionality is slow. Remus has in fact removed XenStore from the
suspend/resume process all together and all seems to function well.
As such, I'm proposing the complete remove of both RELEASE and RESUME
from the XenStore specification.
Additionally, I see a recent modification to XenStore to include a
SET_TARGET command which has plenty of issues. For example, say you have
two domains A and B, both of which have permissions to a node X. A has
full permissions and B only has read permissions. If you set a target
from A to B (or whichever way around it is so that A gets B's
permissions too) and B comes first in the permission list for X, then A
will get B's permissions (read-only) instead of having full permissions.
Thus, SET_TARGET can actually cripple the access of a domain.
This sort of functionality should be able to be implemented without
having a special, dedicated function (through the use of pluggable
policy modules, as mentioned above).
Thanks,
Patrick Colp
Following is a proposed update to the XenStore Protocol Specification. I
have some questions/comments about the protocol, which are the lines
starting with %. I would especially appreciate feedback about those.
XenStore Protocol Specification
-------------------------------
XenStore implements a map between `keys' (which are filename-like
pathnames) and values. Clients may read and write values, watch for
changes, and set permissions to allow or deny access. There is also
a rudimentary transaction system.
% Avoided? Or make it a hard requirement?
% Normally 7-bit ASCII or always?
% Generally add a nul byte? Should they always? Never?
While XenStore and most tools and APIs are capable of dealing with
arbitrary binary data as values, this should generally be avoided.
Instead, data should generally be human-readable for ease of
management and debugging. XenStore is not a high-performance facility
and should be used only for small amounts of control plane data.
Therefore, XenStore values should be 7-bit ASCII text strings
containing bytes 0x20..0x7F only, and should not contain a trailing
nul byte (the APIs used for accessing XenStore generally add a nul
when reading, for the caller's convenience).
Paths are separated by a / and the root path is /, just like in Unix
file systems. A path (<parent>) is a parent of another path (<child>)
if <parent> and <child> are not identical and if <parent> is / (the
root path) or <parent>/ is an initial substring of <child>.
% Conventional to not store values? Should this be a requirement or a
non-convention?
If a path exists, all of its parents do too. Every path maps to a
value, which may be empty. It can also have zero or more immediate
children. There is thus no particular distinction between
directories and leaf nodes. However, it is conventional to not
store values at nodes which also have children.
The permitted character for paths set are the ASCII alphanumerics
and the four punctuation characters -/_@ (hyphen slash underscore
atsign). @ should be avoided except to specify special watches
(see below). Doubled slashes and trailing slashes (except to specify
the root) are forbidden. The empty path is also forbidden. Paths
longer than 3072 bytes are forbidden; clients specifying relative
paths should keep them to within 2048 bytes (see XENSTORE_*_PATH_MAX
in xs_wire.h).
Communication with XenStore is either via sockets or event channel
and shared memory, as specified in io/xs_wire.h. Each message in
either direction has a header formatted as a a struct xsd_sockmsg,
which has the following format:
unsigned 32-bit integer: type
unsigned 32-bit integer: req_id
unsigned 32-bit integer: tx_id
unsigned 32-bit integer: len
After the header, the message contains len bytes of payload.
The payload syntax varies according to the type field. Generally
requests each generate a reply with an identical type, req_id, and
tx_id. However, if an error occurs, a reply will be returned with
type ERROR, and only req_id and tx_id copied from the request.
A caller who sends several requests may receive the replies in any
order and must use req_id (and tx_id, if applicable) to match up
replies to requests.
% Payload is limited to 4096? Or header + payload?
The payload length (len field of the header) is limited to 4096 bytes
(XENSTORE_PAYLOAD_MAX) in both directions. If a client exceeds the
limit, its XenStore connection will be immediately killed by
XenStore, which is usually catastrophic from the client's point of
view. Clients (particularly domains, which cannot just reconnect)
should avoid this.
Due to this limitation, bulk data should not be passed through XenStore
as the performance properties are poor. In addition, this would violate
the intended use of XenStore.
---------- Xenstore Protocol Details - Introduction ----------
The payload syntax and semantics of the requests and replies are
described below. In the payload syntax specifications the following
notations are used:
| A nul (zero) byte.
<foo> A string guaranteed not to contain any nul bytes.
<foo|> Binary data (which may contain zero or more nul bytes)
<foo>|* Zero or more strings each followed by a trailing nul
<foo>|+ One or more strings each followed by a trailing nul
? Reserved value (may not contain nuls)
?? Reserved value (may contain nuls)
Reserved values for the most part will be empty strings. However, they
exist in order to enable extensions in the future.
Error replies are as follows:
ERROR E<something>|
Where E<something> is the name of an errno value
listed in io/xs_wire.h. Note that the string name
is transmitted, not a numeric value.
Where no reply payload format is specified below, success responses
have the following payload:
OK|
Values commonly included in payloads include:
<path>
Specifies a path in the hierarchical key structure.
If <path> starts with a / it simply represents that path.
<path> is allowed to not start with /, in which case the
caller must be a domain (rather than connected via a socket)
and the path is taken to be relative to /local/domain/<domid>
(e.g., `x/y' sent by domain 3 would mean `/local/domain/3/x/y').
<domid>
Integer domid, represented as decimal number 0..65535
(16-bit unsigned integer). Parsing errors and values
out of range generally go undetected. The special
DOMID_... values (see xen.h) are represented as 16-bit
unsigned integers; unless otherwise specified it is
an error not to specify a real domain id.
The following sections give the actual type values, including the
request and reply payloads as applicable.
---------- Database Read, Write, and Permissions Operatons ----------
READ <path>| <value|>
WRITE <path>|<value|>
Store and read the octet string <value> at <path>.
WRITE creates any missing parent paths with empty values.
MKDIR <path>|
Ensures that the <path> exists, if necessary by creating
it and any missing parents with empty values. If <path>
or any parent already exists, its value is left unchanged.
RM <path>|
Ensures that the <path> does not exist, by deleting
it and all of its children. It is not an error if <path> does
not exist, but it _is_ an error if <path>'s immediate parent
does not exist either.
DIRECTORY <path>| <child-leaf-name>|*
Gives a list of the immediate children of <path> as only the
leafnames. The resulting children are each named
<path>/<child-leaf-name>.
GET_PERMS <path>| <perm-as-string>|+
SET_PERMS <path>|<perm-as-string>|+?
<perm-as-string> is one of the following
w<domid> write only
r<domid> read only
b<domid> both read and write
n<domid> no access
See http://wiki.xensource.com/xenwiki/XenBus section
`Permissions' for details of the permissions system.
---------- Watches ----------
WATCH <wpath>|<token>|?
Adds a watch.
When a path is modified (including path creation, removal,
contents change or permissions change) this generates an event
on the changed path. Changes made in transactions cause an
event only if and when committed. Each occurring event is
matched against all the watches currently set up and each
matching watch results in a WATCH_EVENT message (see below).
The event's path matches the watch's <wpath> if it is <wpath>
or a child of <wpath>.
<wpath> can be a path to watch or @<wspecial>. In the
latter case <wspecial> may have any syntax but it matches
(according to the rules above) only the following special
events which are invented by XenStore:
@introduceDomain occurs on INTRODUCE
@releaseDomain occurs on any domain crash or
shutdown, and also on RELEASE
and domain destruction
When a watch is first set up, it is triggered once straight
away, with the path equal to <wpath>. Watches may be triggered
spuriously. The tx_id in a WATCH request is ignored.
Watches are restricted by the permissions system. Applications
will not be sent a notification for paths that they cannot
read. However, an application will be sent a watch when a path
which it is able to read is deleted, even if that leaves only
a nonexistent, unreadable parent. A notification will be
omitted if a node's permissions are changed so as to make it
unreadable, in which case future notifications will also be
suppressed (and if the node is later made readable, some
notifications may have been "lost").
WATCH_EVENT <epath>|<token>|
Unsolicited `reply' generated for matching modfication events
as described above. req_id and tx_id are both 0.
<epath> is the event's path (i.e. the actual path that was
modifed). However, if the event was the recursive removal of
a parent of the watched path, <epath> is the watched path
(rather than the actual path which was removed). So <epath>
is either the watched path or a child of the watched path.
Iff the watched path was specified as a relative pathname,
then <epath> will also be relative (with the same base as
the watched path).
UNWATCH <wpath>|<token>|?
Remove a watch placed on the path <wpath>.
---------- Transactions ----------
TRANSACTION_START | <transid>|
<transid> is an opaque unsigned 32-bit integer allocated by
XenStore. After this, transaction may be referenced by using
<transid> in the tx_id request header field. It is not legal
to send a non-0 tx_id in TRANSACTION_START.
TRANSACTION_END T|
TRANSACTION_END F|
tx_id must refer to and existing transaction. After this
request, the tx_id is no longer valid and may be reused by
XenStore. If F is sent, the transaction is discarded. If T
is sent, it is committed. If T is sent but there were
intervening writes which conflict (meaning only writes or
other commits which changed paths which were read or written
in the transaction at hand), then the writes cause an EAGAIN
message to be sent as a reply.
---------- Domain Management and XenStore Communications ----------
INTRODUCE <domid>|<mfn>|<evtchn>|?
Notifies XenStore to communicate with this domain.
INTRODUCE is used during domain startup, restore, and resume.
<domid> must be a real domain id (not 0 and not a special
DOMID_... value). <mfn> must be a machine page in that domain
represented as an unsigned 32-bit integer. <evtchn> must be an
unbound event channel in the <domid> domain (likewise in 32-bit
unsigned integer), on which XenStore will call bind_interdomain.
XenStore prevents the use of INTRODUCE other than by Domain-0.
RELEASE <domid>|
Manually requests that XenStore disconnect from the domain.
The event channel is unbound at the XenStore end and the
machine page unmapped. If the domain is still running, it
won't be able to communicate with XenStore. Note that XenStore
will in any case detect domain destruction and disconnect by
itself. XenStore prevents the use of RELEASE other than by
Domain-0.
GET_DOMAIN_PATH <domid>| <path>|
Returns the domain's base path, as is used for relative
transactions (i.e. /local/domain/<domid> -- with <domid>
normalised). The answer will be useless unless <domid> is a
real domain id.
IS_DOMAIN_INTRODUCED <domid>| T| or F|
Returns T if XenStore is in communication with the domain
(i.e. if INTRODUCE for the domain has not yet been followed by
domain destruction or explicit RELEASE).
RESUME <domid>|
Arranges that @releaseDomain events will once more be
generated when the domain becomes shut down. This might have
to be used if a domain were to be shut down (generating one
@releaseDomain) and then subsequently restarted, since the
state-sensitive algorithm in XenStore will not otherwise send
further watch event notifications if the domain were to be
shut down again.
It is not clear whether this is possible since one would
normally expect a domain not to be restarted after being shut
down without being destroyed in the meantime. There are
currently no users of this request in xen-unstable.
XenStore prevents the use of RESUME other than by Domain-0.
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home