Re: [Xen-devel] [OSSTEST PATCH] README: Better documentation of recipes, db, etc.

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 09/08/2017 02:52 PM, Ian Jackson wrote:
> CC: George Dunlap <george.dunlap@xxxxxxxxxx>
> Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>

This is a significant improvement, thanks:

Reviewed-by: George Dunlap <george.dunlap@xxxxxxxxxx>



> ---
>  README | 75 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
>  1 file changed, 60 insertions(+), 15 deletions(-)
> 
> diff --git a/README b/README
> index ffe0018..93129e3 100644
> --- a/README
> +++ b/README
> @@ -8,32 +8,55 @@ Terminology
>  
>  "flight":
>  
> -    Each run of osstest is referred to as a "flight". Each flight is
> -    given a unique ID (a number or name).
> +    Each run of osstest is referred to as a "flight".  Each flight is
> +    given a unique ID.
> +
> +    Standalone mode generally uses a flight named "standalone", which
> +    is frequently erased and reused.  In "Executive" mode (used for
> +    production osstest instances) flights are numbered and the
> +    metadata is permanently recorded.
>  
>  "job":
>  
> -    Each flight consists of one or more "jobs". These are a sequence
> +    Each flight consists of one or more "jobs".  These are a sequence
>      of test steps run in order and correspond to a column in the test
> -    report grid. They have names like "build-amd64" or
> -    "test-amd64-amd64-pv". A job can depend on the output of another
> +    report grid.  They have names like "build-amd64" or
> +    "test-amd64-amd64-pv".  A job can depend on the output of another
>      job in the flight -- e.g. most test-* jobs depend on one or more
>      build-* jobs.
>  
> +    A job has a named "recipe", which indicates what things need to be
> +    done in what order: generally, a sequence of ts-* scripts.  The
> +    recipes are defined by their implementations in sg-run-job.
> +
> +    The set of jobs for any particular kind of flight is defined in
> +    make-flight, make-*-flight, and mfi-*.
> +
>  "step":
>  
> -    Each job consists of multiple "steps" which is an individual test
> -    operation, such as "build the hypervisor", "install a guest",
> -    "start a guest", "migrate a guest", etc. A step corresponds to a
> -    cell in the results grid. A given step can be reused in multiple
> -    different jobs, e.g. the "xen build" step is used in several
> -    different build-* jobs. This reuse can be seen in the rows of the
> -    results grid.
> +    Running a job consists of running its individual "steps".  Each
> +    step is an individual test operation, such as "build the
> +    hypervisor", "install a guest", "start a guest", "migrate a
> +    guest", etc.
> +
> +    Generally a step corresponds to one execution of some ts-* script.
> +
> +    steps have a "testid" which is used to uniquely identify the same
> +    operation in different test runs.  This is used for identifying
> +    regressions, bisection, and so on.
> +
> +    The steps for a job, including the testids, are defined by the
> +    recipe in sg-run-job (see "job", above).
> +
> +    Each step run in a job becomes a cell in the HTML results grid.  A
> +    given step might appear in multiple different jobs, e.g. the "xen
> +    build" step is used in several different build-* jobs.  This can
> +    be seen in the rows of the results grid.
>  
>  "runvar":
>  
> -    A runvar is a named textual variable associated with each job in a
> -    given flight. They serve as both the inputs and outputs to the
> +    A runvar is a named textual variable associated with a given job
> +    and flight. They serve as both the inputs and outputs to the
>      job.
>  
>      For example a Xen build job may have input runvars "tree_xen" (the
> @@ -55,6 +78,15 @@ Terminology
>          * the parameters of the guest to test (e.g. distro, PV vs HVM
>            etc).
>  
> +    Runvar names often have structure, being assembled out of various
> +    pieces; and the names are sometimes parsed, too.
> +
> +flights, jobs and runvars are kept in the database (in standalone
> +mode, "standalone.db").  As for steps, only steps which have been
> +started are recorded in the db; steps which have yet to be attempted
> +are not in the database.  Also, if a ts-* script is run by hand, this
> +is not recorded as a step.
> +
>  Operation
>  =========
>  
> @@ -73,12 +105,25 @@ referenced by each job's configuration. It then runs 
> each of these in
>  turn, taking into account the prerequisites etc, by calling the
>  relevant "ts-*" scripts.
>  
> +Each ts-* is a convenient unit of test execution and reporting.  Most
> +ts-* scripts are written in perl, although simple shell wrappers are
> +sometimes used.  ts-* scripts expect OSSTEST_FLIGHT and OSSTEST_JOB to
> +be set in the environment.  Most of them also expect to be told extra
> +information on the command line - notably, which host(s) to operate
> +on.
> +
> +In automatic operation, the command line arguments for each test
> +step's ts-* script invocation are fixed by the recipe defined in
> +sg-run-job.
> +
>  When running in standalone mode it is possible to run any of these
>  steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although
>  you will need to find the correct inputs (some of which are documented
>  below) and perhaps take care of prerequisites yourself (e.g. running
>  "./sg-run-job test-armhf-armhf-xl" means you must have done
> -"./sg-runjob build-armhf" and "build-armhf-pvops" first.
> +"./sg-runjob build-armhf" and "build-armhf-pvops" first.  When running
> +a ts-* script manually one often wants to specify "host=<hostname>".
> +(See the head comment for "selecthost" in Osstest/TestSupport.pm.)
>  
>  Results
>  =======
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel