Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL

[Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>]

Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions 
- FAIL"):
> On Fri, 2015-11-27 at 01:18 -0700, Jan Beulich wrote:
> > Neither of these failed in this flight, and there's nothing else blocking
> > the push. Why did this not result in a push then? Or in other words
> > why do the failures in earlier flights get considered a reason not to
> > push?
> 
> @Ian, README.email covers lots of these kinds of patterns, but not this
> specific one.

See below for proposed docs patch to explain the general meaning of
`fail in X REGR. vs. Y'.


> > >  build-i386 5 xen-build fail in 65062 REGR. vs. 63449

This is completely explained below, I think.

> > >  test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 16 
> > > guest-localmigrate/x10 fail in 65088 REGR. vs. 63449

As explained below, in 65112 this step did not run because the earlier
step `guest-localmigrate' failed:
  
http://logs.test-lab.xenproject.org/osstest/logs/65112/test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html

The fact that we have both `guest-localmigrate' and
`guest-localmigrate/x10' isn't ideal because it hides from the
heisenbug compensator that these are actually the same underlying
test.  Maybe it is time now to rename `guest-localmigrate/x10' to
`guest-localmigrate' and abolish the latter.


From 987dd088192f9f94c59beeddc073cecaad76a24e Mon Sep 17 00:00:00 2001
From: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
Date: Fri, 27 Nov 2015 11:36:05 +0000
Subject: [OSSTEST PATCH] README.email: Document `fail in 58948 REGR. vs.
 63449'

Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
---
 README.email |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/README.email b/README.email
index 992a574..40df71a 100644
--- a/README.email
+++ b/README.email
@@ -71,6 +71,24 @@ history.  Here are some examples:
       detect regressions of this test.  Perhaps the test has been
       recently introduced.
 
+   fail in 58948 REGR. vs. 63449
+
+      The results processor used 58948 (another flight testing the
+      just-tested version) to convince itself that some other test
+      failure is intermittent.  Look for other references to 58948 in
+      the report to see which those other test failures are.
+
+      However, in 58948, there were further failures.  In particular,
+      the step being reported here failed, and that failure could not
+      in turn be justified.
+
+      If this further failure is in a test job, this is usually
+      because the reported step did not run at all in the most recent
+      flight, usually because it was blocked by an earlier failure.
+      (Intermittent build job failures are never considered
+      justifiable because they prevent other tests from running and
+      can so conceal bugs.)
+
    fail in 58948 pass in 58965
    fail in 58948 like 37628
 
-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel