Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL

[Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>]

Jan Beulich writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions 
- FAIL"):
> On 27.11.15 at 13:02, <Ian.Jackson@xxxxxxxxxxxxx> wrote:
> > The fact that we have both `guest-localmigrate' and
> > `guest-localmigrate/x10' isn't ideal because it hides from the
> > heisenbug compensator that these are actually the same underlying
> > test.  Maybe it is time now to rename `guest-localmigrate/x10' to
> > `guest-localmigrate' and abolish the latter.
> 
> Independent of that, does it make sense for a dependent test to
> not be considered failing intermittently when the test it depends on
> is?

Suppose two test steps A and B, which normally run in that order.
Suppose failure of A prevents the execution of B (this is the usual
case where step A precedes step B; normally later steps in a job
depend on the success of earlier steps, because after an earlier
failure the testbed state is not necessarily well-defined).

Now suppose A has an intermittent bug, but B is totally broken.

With our current policy on intermittent bugs[1], we would allow a push
despite the bug in A.  But we should not allow a push despite B: the
100% reproducible failure of B should prevent all pushes.

But the bug in B only shows up when A happens to pass.  So the
heisenbug compensator has to insist on seeing an actual pass of B
(which in this hypothetical situation, will not occur).

Eg, consider these flights:

  100  is now master  A pass, B pass      pushed
  200  staging        A pass, B fail      `B REGR. vs 100'
  201  staging        A fail, B not run   `B fail in 200 REGR. vs 100'

In flight 201, the failure of A is indeed justifiable as a heisenbug
because it can be seen to succeed in flight 200.  It is the problem
with B which is actually blocking the push - it is merely that the
failure occurred in flight 200.

If, contrary to my suppositions above, the failure of B is actually a
heisenbug, then hopefully eventually both A and then B will happen to
pass in the same run.  Even if that particular flight has other
problems, a future evaluation of a test of the same version can use
that flight's passes of A and B to justify, respectively, whatever
failures of A and/or B that it comes across.

Ian.

[1] In principle we could have a different policy: to try to reject
intermittent bugs.  But it would require a lot of test resources
because all tests would have to be repeated a lot, and naturally
intermittent bugs would slip through anyway.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel