Re: [Xen-devel] How does the regression testing stuff work?

[Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>]

Alex Bligh writes ("[Xen-devel] How does the regression testing stuff work?"):
> So this came in on 5th April (latest example of a regression
> message for qemu-upstream-4.2-testing). I would take this to
> mean the changesets applied (the two listed below) caused
> a regression in the test guest-localmigrate/x10 (whatever
> that is).

Yes.  Although the apparent regression appeared once in this one test,
so if the tree is unreliable (sadly it seems to be) it could be a
heisenbug.

> However, the two commits did get pushed from staging to the
> main repos, with two subsequent 'tolerable FAIL - pushed' messages.

Right.

> I'm not the author of the two commits in question, but if I
> was I'd want to know what test actually failed and why. The logs
> weren't vastly illuminating without knowing what the test does.
> I'd also be keen to know whether I should be investigating this
> or whether waiting a couple of days might result in a retest
> that appears to pass the code.

If you visit this url:

> flight 17485 qemu-upstream-4.2-testing real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/17485/

You'll see a big grid.

> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-amd64-xl-qemuu-winxpsp3 12 guest-localmigrate/x10 fail REGR.
> vs. 17143

Look at the column for the failure.  You can see it because of the
red/pink REGR entry in the grid, or you can look at the column
heading.

Click on the column heading.

This will show you a lot of logs.  The one which is most relevant is
the test script log, which is linked to from "fail" in the Steps table
there.

http://www.chiark.greenend.org.uk/~xensrcts/logs/17485/test-amd64-amd64-xl-qemuu-winxpsp3/12.ts-guest-localmigrate.log

So you can see there that the tester ran the 9th repetition of

  ssh ... root@xxxxxxxxxxxx xl migrate win.guest.osstest localhost

and that appeared to work in the sense that it produced sane-looking
output, but it didn't exit.  So 400s later the tester says this:

 ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o
 ConnectTimeout=100 -o ServerAliveInterval=100 -o
 PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o
 UserKnownHostsFile=tmp/t.known_hosts_17485.test-amd64-amd64-xl-qemuu-winxpsp3
 root@xxxxxxxxxxxx xl migrate win.guest.osstest localhost
 status (timed out) at Osstest/TestSupport.pm line 375.

I don't know why this didn't work, but if it passed later it must be a
race of some kind.  It therefore may be the case that the changes
being tested were not in fact responsible.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel