Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> On 04/08/16 12:49, Ian Jackson wrote:
> > Andrew, do you want to have osstest run its xtf push gate between
> > `staging' and `master' branches of xtf, or do you want to just push to
> > xtf master, and have a separate `osstest/tested' or some such branch
> > for the osstest push gate output ?
> 
> The latter please.

NP.

> We should also clarify the force push criteria.  It is moderately likely
> that we get a fix or extension to an existing test which starts showing
> up a new bug in the code under test.

Can this not be made into a new test ?

osstest would like such an extension to an existing test to be a new
`step' (in osstest terminology).  That would mean that the fact that
it fails with old versions of Xen can be seen not to be a regression.

Ie, osstest would like to be able to distinguish:
  * this version of xtf didn't have the new part of the test
  * old part of the test failed (and presumably new part was not run)
  * old part succeeded and new part failed
  * both parts were run and succeeded

> In this case, OSSTest will identify a regression, but it isn't a
> regression isn't in XTF, nor is it reasonable to call it a regression in
> Xen.  We would absolutely want to fix it in upstream, but there are no
> guarentees that the bugfix would be suitable for backport to older trees.

Indeed.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel