Re: Proposal - lightweight auto test binaries

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Wed, Jan 13, 2021 at 03:45:10PM +0000, Andrew Cooper wrote:
> On 12/01/2021 17:35, Ian Jackson wrote:
> >  * When appropriately configured, the xen.git build system
> >    will ship them into dist/install/usr/local/....
> >
> >  * There will be a specific directory containing tests which
> >    are suitable for running automatically:
> >       dist/install/usr/local/lib/xen/autotests
> >
> >  * The pass/success reporting API will be encoded into the
> >    filename.  For now we define only one API:
> >      dist/install/usr/local/lib/xen/autotests/simple/*
> 
> Whatever we go for, it ought to be an easy path, and honestly - that's a
> mouthful to get to.  These tests need to be just as easy for developers
> to use.
> 
> How about opt/xen-autotest/* ?  Much easier in a deb/rpm build to filter
> the things which clearly shouldn't be in a production system.  It's also
> invariant of libdir which is an advantage for downstreams who test on
> more than one distro.

I can see them being packaged as a separate sub-package instead of not
packaging at all. And many distributions have rules forbidding native
packages form putting stuff into /opt. So a location somewhere in /usr
would be preferable.
I'm fine with dist/install/usr/local/lib/xen/autotests.
For running manually, there could be a trivial test runner (shell
script) that could be installed into dist/install/usr/local/sbin - to
make it even more convenient to access.

> I certainly have plans to reuse this infrastructure in XenRT.  I expect
> other downstreams would as well.
> 
> > The "simple" test API is as follows:
> >
> >  * Every executable file (or symlink to one) in
> >      dist/install/usr/local/lib/xen/autotests/simple/
> >    will be run with no arguments.
> >
> >  * Only files whose names contain only [-0-9a-z] will be run.
> >    Others will be ignored.  NB _ is not allowed.
> 
> I'm happy not using _, but why the restriction?
> 
> >  * A test may exit with one of the XTF exit statuses:
> >           0 # XTF SUCCESS
> >           3 # XTF SKIP
> >           4 # XTF ERROR
> >           5 # XTF FAILURE
> >           6 # XTF CRASH
> >    (osstest will treat anything other than 0 and 3 as "fail")
> >    Any other termination should be treated as CRASH.
> >
> >  * If a test exits nonzero it should print some information about
> >    the situation to stderr.
> >
> >  * Tests may print other information to stderr, which will be captured
> >    and logged.
> >
> >  * Tests *must not* print anything to stdout.  Test runners *must*
> >    ignore stdout (though they may log it, possibly mixed in with
> >    stderr).  stdout is reserved for future extension.
> 
> I disagree.  It is far more important that the tests are pleasant for
> humans to use, both in terms of developing them to begin with, and
> maintaining them if they go wrong.

I can see why such restriction would be useful for future extensions,
but I don't think it's strictly necessary. An extension can for example
require stdout lines with a specific prefix and ignore others.

> In practice, this means human-useful stuff on stdout.  The test
> automation does not need to care - all it should do is log
> stdout/stderr, and act upon the exit code.  (Perhaps this is what you
> meant, but its not clear IMO.)
> 
> 
> >  * Tests should exit with XTF ERROR if they are passed any arguments.
> 
> ./test-foo --debug-this-harder would be a perfectly reasonable thing for
> a human to use to investigate problems, and it is what we use in the
> fuzzing harnesses for investigating crashes.

A data point: libvirt test suite have very same requirement - test
binaries needs to be run without an argument and error when they get
one. In addition to the error, you get a help message with env variables
to use instead (VIR_TEST_DEBUG=1 for example). And I do run it with some
dummy argument, to get this help message, because I can't remember how
those variables are named. Every single time.
But TBH, if parameters wouldn't be very obvious (like --verbose, or
--debug), I'd run --help first too.

In any case, unknown command like arguments should result in an error.

> The problem is "for these configurations".  Even at the totally basic level,
> 
> * PV and Shadow are conditional on Xen's Kconfig.
> * PV32 is further conditional on command line settings, and/or whether
> the hardware supports CET-SS.
> * HVM is dependent on Kconfig, hardware, firmware and command line
> settings.  HAP similarly.
> 
> `xl create` doesn't handle missing CONFIG_SHADOW, or PV32 being disabled
> cleanly, despite having suitable information available in `xl info`. 
> While this can (and should) be fixed, its not helpful for the more
> general testing case.

It indeed is an issue, but in practice this can be worked up
incrementally - when the test fails, see if the reason is unsupported
(by the test) configuration. And if so - add a condition for this
specific case (to return SKIP or choose another configuration). Rinse
and repeat.

> At the moment, some of this information can be obtained from Xen, but
> the details are scattered through various stable and unstable
> hypercalls, and in at least one case, in a stringly typed
> datastructure.  None are pleasant to obtain at the shell level.
> 
> Instead, I was wondering if we should come up with some stable hypfs
> layout to report Xen platform capabilities.  e.g.
> 
> platform/pv/ (dir)
> platform/pv/abis = "xen-3.0-x86_64 xen-3.0-x86_32p" (string, but maybe
> we drop this now as obsolete)
> platform/pv/32 (bool or dir)
> platform/hvm/ (dir)
> platform/hvm/shadow (bool)
> platform/hvm/hap (bool)
> 
> Fundamentally, we care about *what* xen can/cannot do in this
> configuration on this hardware, far more than we care about specifically
> *why* Xen can't do something[0].

Yes, I very much agree. But I think it's only slightly related (and
definitely not a blocker) for this test layout/API.

> These can be retrieved in a reasonably convenient manner on the shell
> with $(xenhypfs cat /path) and we could definitely do something more
> useful for scripting with exit codes, etc.
> 
> When it comes back to testing, we will have a binary which has a range
> of configurations built in, and need some mechanism to reliably
> calculate the subset of configurations applicable in the current situation.
> 
> However, again in the interest of making the tests useable for humans, I
> might go as far as to suggest that we have two separate binaries, a
> `$FOO-select` which prints out a list of suitable list of `$FOO
> --config-$X` to run and treat as independent "simple" tests.

I'd simply go with separate tests returning a SKIP if not applicable on
particular config. Internally of course it can be the same code that
gets called with different parameters.
In Qubes we do that (various tests that are run on every VM, and some
applicable only for Debian/Fedora return a SKIP on the other one) and
it works pretty well.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Attachment: signature.asc
Description: PGP signature