Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 04/08/16 17:13, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf 
> tree"):
>> On 04/08/16 16:21, Ian Jackson wrote:
>>> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf 
>>> tree"):
>>>> We should also clarify the force push criteria.  It is moderately likely
>>>> that we get a fix or extension to an existing test which starts showing
>>>> up a new bug in the code under test.
>>> Can this not be made into a new test ?
>> That very much depends, and probably needs deciding on a case by case basis.
>>
>> What absolutely shouldn't happen is ending up with test-$FOO,
>> test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
>> the same logical test being split up in ad-hoc ways.  While XTF is
>> useful for automation, it is first and foremost a tool for humans.
> You mean that this would contort the code for the test ?
>
> Here is another approach that could be used to satisfy osstest's
> desire for stability in the meaning of test names: you could
> explicitly copy the entire source of the test to a new test name, and
> deprecate the old test.  Later, if the old versions of Xen are fixed,
> or after they are retired you would delete the old test.
>
> The result would be that the old, less-good, test would still exist
> and still detect regressions.  The new, better test, would be seen to
> have never passed on old branches.
>
> Force pushing XTF is not an answer to this problem, for two reasons:
>
> Firstly, if the improved test correctly fails on old versions of Xen,
> this will appear as a regression in each old version of Xen.  Each old
> Xen tree would need to be force pushed, every time this happened.
>
> Secondly, in any case, force pushing XTF might not be necessary.  In
> the usual case, one would expect xen-unstable (which is what would
> drive the XTF push gate) to be fixed so that the improved test
> passes.  So osstest wouldn't spot the issue until it ran the new XTF
> with old Xen.
>
> Another possibility would be to have the capability for an single XTF
> test execution to return multiple statuses.

After an IRL discussion, the following has been proposed.

Each test can have a revision number associated with it.  The revision
gets bumped every time there is a change to the test which adds new
functionality (including bugfixes which cause it to spot a preexisting
error).

By default, a human will just want the latest and greatest.  Test
systems on the other hand will want to test "the same thing as before".

The XTF build system will support a way to build older revisions of the
tests.  It is likely that this will involve recompiling the current code
with an older revision id, and leave the onus on the programmer to
ensure that the result is functionally equivalent to the older test.

This way, test automation systems can internal treat different revisions
of the test as logically separate, for the purpose of identifying
regressions.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel