Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-11-27 at 13:24 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112:
> regressions - FAIL"):
> > On Fri, 2015-11-27 at 12:02 +0000, Ian Jackson wrote:
> > > As explained below, in 65112 this step did not run because the
> > > earlier
> > > step `guest-localmigrate' failed:
> > > Â http://logs.test-lab.xenproject.org/osstest/logs/65112/test-amd64-
> > > amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html
> > 
> > Would it be possible to arrange for "blocked" to appear somewhere in
> > the
> > results for the job? e.g. "blocked fail in XXX REGR. vs. YYY".
> > README.email
> > says "The results normally start with the result in this flight" and I
> > think this would be in keeping with that.
> 
> But it might not be true that it was blocked.

Can't sg-run-job tell if it was blocked vs something else though?

I was suggesting to only add blocked if it was blocked, I'm not sure what I
was suggesting to do for other reason not to run, because I hadn't really
considered it, but those would be unusual I think?

> ÂÂMaybe the version of
> osstest used didn't have that step at all, for example.

In which case would it still be considering the step for failures at all?

i.e. if:

flight 100 had test-foo == pass
flight 200 had test-foo == fail (blocking)
flight 201 had test-foo == blocked; fail in 201 vs 100
flight 202 had no test-foo present at all

Would the decision for flight 202 really be to consider the test-foo
results in 100, 200 and 201, and therefore block?

> The best you could say would be something like
> Â "not run; fail in XXX REGR. vs. YYY"
> but that poses more questions than it answers.

Right.

> 
> > Otherwise I think people naturally tend to just read the "and are
> > blocking"
> > section and forget to consider that non-blocking stuff further down may
> > have (tolerably) failed but then blocking something else which is then
> > blocking the push.
> 
> Perhaps sg-report-flight could, if there are any blockages of the form
> `fail in XXX REGR. vs YYY', add a note below the blockage section,
> saying something like `XXX examined since needed to justify other
> failures, see below'.
> 
> I'm a bit reluctant to suggest this because it is, essentially,
> boilerplate - it would always say the same thing about any `fail in
> XXX' - and filling reports like this with boilerplate isn't always a
> good idea.

In general I agree, in this case it might be worth it to counteract a
(perfectly understandable IMHO) natural tendency to only look at the
section labelled blocking, it's basically "don't forget that this non-
blocking stuff might actually be relevant to the blockage".

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel