Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 04/08/16 16:21, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf 
> tree"):
>> On 04/08/16 12:49, Ian Jackson wrote:
>>> Andrew, do you want to have osstest run its xtf push gate between
>>> `staging' and `master' branches of xtf, or do you want to just push to
>>> xtf master, and have a separate `osstest/tested' or some such branch
>>> for the osstest push gate output ?
>> The latter please.
> NP.
>
>> We should also clarify the force push criteria.  It is moderately likely
>> that we get a fix or extension to an existing test which starts showing
>> up a new bug in the code under test.
> Can this not be made into a new test ?

That very much depends, and probably needs deciding on a case by case basis.

What absolutely shouldn't happen is ending up with test-$FOO,
test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
the same logical test being split up in ad-hoc ways.  While XTF is
useful for automation, it is first and foremost a tool for humans.

~Andrew

>
> osstest would like such an extension to an existing test to be a new
> `step' (in osstest terminology).  That would mean that the fact that
> it fails with old versions of Xen can be seen not to be a regression.
>
> Ie, osstest would like to be able to distinguish:
>   * this version of xtf didn't have the new part of the test
>   * old part of the test failed (and presumably new part was not run)
>   * old part succeeded and new part failed
>   * both parts were run and succeeded
>
>> In this case, OSSTest will identify a regression, but it isn't a
>> regression isn't in XTF, nor is it reasonable to call it a regression in
>> Xen.  We would absolutely want to fix it in upstream, but there are no
>> guarentees that the bugfix would be suitable for backport to older trees.
> Indeed.
>
> Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel