[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [xen-unstable test] 2058: regressions - FAIL



On Wed, 2010-09-01 at 10:01 +0100, xen.org wrote:
> flight 2058 xen-unstable real
> http://www.chiark.greenend.org.uk/~xensrcts/logs/2058/
> 
> Regressions :-(
> 
> tests which did not succeed:
>  test-amd64-amd64-pair        11 guest-start                fail REGR. vs. 
> 2053

        2010-09-01 07:33:18 Z executing ssh ... root@xxxxxxxxxxxxx xm create 
/etc/xen/debian.guest.osstest.cfg
        Warning: Permanently added '10.80.249.102' (RSA) to the list of known 
hosts.
        Using config file "/etc/xen/debian.guest.osstest.cfg".
        Started domain debian.guest.osstest (id=1)
        2010-09-01 07:33:21 Z executing ssh ... root@xxxxxxxxxxxxx xm list
        Warning: Permanently added '10.80.249.102' (RSA) to the list of known 
hosts.
        2010-09-01 07:33:21 Z guest debian.guest.osstest state is r
        2010-09-01 07:33:21 Z guest debian.guest.osstest 5a:36:0e:0a:00:10 22 
link/ip/tcp: waiting 100s...
        2010-09-01 07:33:21 Z guest debian.guest.osstest 5a:36:0e:0a:00:10 22 
link/ip/tcp: no entry in statedb::ips (waiting) ...
        guest debian.guest.osstest 5a:36:0e:0a:00:10 22 link/ip/tcp: wait timed 
out: no entry in statedb::ips.

It would be useful to timestamp this last entry, for a while I was
unsure if we'd actually waited 100s or not.

The guest log ends with:
        [    0.678464]   Magic number: 1:252:3141
        [    0.680053] Freeing unused kernel memory: 600k freed
        [    0.680072] Write protecting the kernel read-only data: 8756k
        Loading, please wait...
        Begin: Loading essential drivers ... done.
        Begin: Running /scripts/init-premount ... done.
        Begin: Mounting root file system ... Begin: Running /scripts/local-top 
...   Volume group "bedbug.cam.xci-test.com" not found
        done.
        Begin: Waiting for root file system ... 

I'm not sure if you said something in the past about the LVM thing being
a benign error due to using the same initrd everywhere (in which case I
think it would be better to arrange for the test suite to generate a
proper matching ramdisk for each kernel/VM, since it is quite confusing
and always sows the seeds of doubt in my mind that a problem might be a
ramdisk configuration issue)

The guest log doesn't appear to show blkfront firing up, although there
is no message about a VBD device without a driver. The xenstore listing
in the logs shows a VBD device in state 4 (connected) as expected, the
backend is blkback (over nbd, I think).

I don't see anything in the domain 0 logs pointing to the cause of the
hang. My guess would be that blkback has hung up reading from the nbd
device, probably while the guest was trying to read the partition table,
hence no messages about the disk in the guest log.

>  test-amd64-amd64-pv           7 guest-start                fail REGR. vs. 
> 2053

Similar pattern to above. Are we confident that the underlying nbd
infrastructure is not flakey?

This specific test case appears to use NBD even though all guests are
local (there's only one host), perhaps we should avoid NBD except where
necessary to avoid spurious failures?

We should probably also add blktap on NFS to the tested combinations too
if it isn't already there, this would also cover the
migration-with-blktap case as well as helping to identify when failures
were NBD related.

Can we add the output of something like
        ps wwwaxf -eo pid,tty,stat,time,nice,psr,pcpu,pmem,nwchan,wchan:25,args
from dom0 to the captured logs so we can see where processes are hung
up. In particular it would have been interesting to know where the qemu
was hung in the windows-install failure below.

Ian.

>  test-amd64-amd64-xl           7 guest-start                fail REGR. vs. 
> 2053
>  test-amd64-i386-pair         11 guest-start                  fail    like 
> 2053
>  test-amd64-i386-pv            7 guest-start                  fail    like 
> 2053
>  test-amd64-i386-xl            7 guest-start                fail REGR. vs. 
> 2053
>  test-amd64-xcpkern-i386-win   5 windows-install            fail REGR. vs. 
> 2053
>  test-amd64-xcpkern-i386-xl   10 guest-stop                   fail   never 
> pass
>  test-i386-i386-pair          11 guest-start                  fail    like 
> 2053
>  test-i386-i386-pv             7 guest-start                  fail    like 
> 2053
>  test-i386-i386-xl             7 guest-start                fail REGR. vs. 
> 2053
>  test-i386-xcpkern-i386-pair  12 guest-migrate/src_host/dst_host fail like 
> 2053
>  test-i386-xcpkern-i386-xl    10 guest-stop                   fail   never 
> pass
> 
> version targeted for testing:
>  xen                  573ddf5cc145
> baseline version:
>  xen                  3c4c3d48a835
> 
> jobs:
>  build-amd64                  pass
>  build-amd64-oldkern          pass
>  build-i386                   pass
>  build-i386-oldkern           pass
>  build-i386-xcpkern           pass
>  test-amd64-amd64-pair        fail
>  test-amd64-amd64-pv          fail
>  test-amd64-amd64-win         pass
>  test-amd64-amd64-xl          fail
>  test-amd64-i386-pair         fail
>  test-amd64-i386-pv           fail
>  test-amd64-i386-win          pass
>  test-amd64-i386-xl           fail
>  test-amd64-xcpkern-i386-pair pass
>  test-amd64-xcpkern-i386-pv   pass
>  test-amd64-xcpkern-i386-win  fail
>  test-amd64-xcpkern-i386-xl   fail
>  test-i386-i386-pair          fail
>  test-i386-i386-pv            fail
>  test-i386-i386-win           pass
>  test-i386-i386-xl            fail
>  test-i386-xcpkern-i386-pair  fail
>  test-i386-xcpkern-i386-pv    pass
>  test-i386-xcpkern-i386-win   pass
>  test-i386-xcpkern-i386-xl    fail
> 
> -------------------------------------------------------------------------------
> build-amd64:
>  1 host-install(1)              pass
>  2 host-build-prep              pass
>  3 xen-build                    pass
>  linux           2968b258b1ca6bd16d75
>  qemu            1734e63891573d4729c7
>  xen             22078:573ddf5cc145
> -------------------------------------------------------------------------------
> build-amd64-oldkern:
>  1 xen-build                    pass
>  linux           1029:9b1adfb8b0b3
>  qemu            1734e63891573d4729c7
>  xen             22078:573ddf5cc145
> -------------------------------------------------------------------------------
> build-i386:
>  1 host-install(1)              pass
>  2 host-build-prep              pass
>  3 xen-build                    pass
>  linux           2968b258b1ca6bd16d75
>  qemu            1734e63891573d4729c7
>  xen             22078:573ddf5cc145
> -------------------------------------------------------------------------------
> build-i386-oldkern:
>  1 xen-build                    pass
>  linux           1029:9b1adfb8b0b3
>  qemu            1734e63891573d4729c7
>  xen             22078:573ddf5cc145
> -------------------------------------------------------------------------------
> build-i386-xcpkern:
>  1 kernel-build                 pass
>  linux           110879:32fc6955a6a5
>  pq_linux        154:61499f9e9a87
> -------------------------------------------------------------------------------
> test-amd64-amd64-pair:
>  1 xen-build-check(1)           pass
>  2 host-install/src_host(2)     pass
>  3 host-install/dst_host(3)     pass
>  4 xen-install/src_host         pass
>  5 xen-install/dst_host         pass
>  6 xen-boot/src_host            pass
>  7 xen-boot/dst_host            pass
>  8 debian-install/dst_host      pass
>  9 debian-fixup/dst_host        pass
> 10 guests-nbd-mirror            pass
> 11 guest-start                  fail
> 12 capture-logs/src_host(12)    pass
> 13 capture-logs/dst_host(13)    pass
> -------------------------------------------------------------------------------
> test-amd64-amd64-pv:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  fail
>  8 capture-logs(8)              pass
> -------------------------------------------------------------------------------
> test-amd64-amd64-win:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 windows-install              pass
>  6 guest-saverestore            pass
>  7 guest-localmigrate           pass
>  8 guest-stop                   pass
>  9 capture-logs(9)              pass
> -------------------------------------------------------------------------------
> test-amd64-amd64-xl:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  fail
>  8 capture-logs(8)              pass
> -------------------------------------------------------------------------------
> test-amd64-i386-pair:
>  1 xen-build-check(1)           pass
>  2 host-install/src_host(2)     pass
>  3 host-install/dst_host(3)     pass
>  4 xen-install/src_host         pass
>  5 xen-install/dst_host         pass
>  6 xen-boot/src_host            pass
>  7 xen-boot/dst_host            pass
>  8 debian-install/dst_host      pass
>  9 debian-fixup/dst_host        pass
> 10 guests-nbd-mirror            pass
> 11 guest-start                  fail
> 12 capture-logs/src_host(12)    pass
> 13 capture-logs/dst_host(13)    pass
> -------------------------------------------------------------------------------
> test-amd64-i386-pv:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  fail
>  8 capture-logs(8)              pass
> -------------------------------------------------------------------------------
> test-amd64-i386-win:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 windows-install              pass
>  6 guest-saverestore            pass
>  7 guest-localmigrate           pass
>  8 guest-stop                   pass
>  9 capture-logs(9)              pass
> -------------------------------------------------------------------------------
> test-amd64-i386-xl:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  fail
>  8 capture-logs(8)              pass
> -------------------------------------------------------------------------------
> test-amd64-xcpkern-i386-pair:
>  1 xen-build-check(1)           pass
>  2 host-install/src_host(2)     pass
>  3 host-install/dst_host(3)     pass
>  4 xen-install/src_host         pass
>  5 xen-install/dst_host         pass
>  6 xen-boot/src_host            pass
>  7 xen-boot/dst_host            pass
>  8 debian-install/dst_host      pass
>  9 debian-fixup/dst_host        pass
> 10 guests-nbd-mirror            pass
> 11 guest-start                  pass
> 12 guest-migrate/src_host/dst_host pass
> 13 guest-migrate/dst_host/src_host pass
> 14 capture-logs/src_host(14)    pass
> 15 capture-logs/dst_host(15)    pass
> -------------------------------------------------------------------------------
> test-amd64-xcpkern-i386-pv:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  pass
>  8 guest-saverestore            pass
>  9 guest-localmigrate           pass
> 10 guest-stop                   pass
> 11 capture-logs(11)             pass
> -------------------------------------------------------------------------------
> test-amd64-xcpkern-i386-win:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 windows-install              fail
>  6 capture-logs(6)              pass
> -------------------------------------------------------------------------------
> test-amd64-xcpkern-i386-xl:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  pass
>  8 guest-saverestore            pass
>  9 guest-localmigrate           pass
> 10 guest-stop                   fail
> 11 capture-logs(11)             pass
> -------------------------------------------------------------------------------
> test-i386-i386-pair:
>  1 xen-build-check(1)           pass
>  2 host-install/src_host(2)     pass
>  3 host-install/dst_host(3)     pass
>  4 xen-install/src_host         pass
>  5 xen-install/dst_host         pass
>  6 xen-boot/src_host            pass
>  7 xen-boot/dst_host            pass
>  8 debian-install/dst_host      pass
>  9 debian-fixup/dst_host        pass
> 10 guests-nbd-mirror            pass
> 11 guest-start                  fail
> 12 capture-logs/src_host(12)    pass
> 13 capture-logs/dst_host(13)    pass
> -------------------------------------------------------------------------------
> test-i386-i386-pv:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  fail
>  8 capture-logs(8)              pass
> -------------------------------------------------------------------------------
> test-i386-i386-win:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 windows-install              pass
>  6 guest-saverestore            pass
>  7 guest-localmigrate           pass
>  8 guest-stop                   pass
>  9 capture-logs(9)              pass
> -------------------------------------------------------------------------------
> test-i386-i386-xl:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  fail
>  8 capture-logs(8)              pass
> -------------------------------------------------------------------------------
> test-i386-xcpkern-i386-pair:
>  1 xen-build-check(1)           pass
>  2 host-install/src_host(2)     pass
>  3 host-install/dst_host(3)     pass
>  4 xen-install/src_host         pass
>  5 xen-install/dst_host         pass
>  6 xen-boot/src_host            pass
>  7 xen-boot/dst_host            pass
>  8 debian-install/dst_host      pass
>  9 debian-fixup/dst_host        pass
> 10 guests-nbd-mirror            pass
> 11 guest-start                  pass
> 12 guest-migrate/src_host/dst_host fail
> 13 capture-logs/src_host(13)    pass
> 14 capture-logs/dst_host(14)    pass
> -------------------------------------------------------------------------------
> test-i386-xcpkern-i386-pv:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  pass
>  8 guest-saverestore            pass
>  9 guest-localmigrate           pass
> 10 guest-stop                   pass
> 11 capture-logs(11)             pass
> -------------------------------------------------------------------------------
> test-i386-xcpkern-i386-win:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 windows-install              pass
>  6 guest-saverestore            pass
>  7 guest-localmigrate           pass
>  8 guest-stop                   pass
>  9 capture-logs(9)              pass
> -------------------------------------------------------------------------------
> test-i386-xcpkern-i386-xl:
>  1 xen-build-check(1)           pass
>  2 host-install(2)              pass
>  3 xen-install                  pass
>  4 xen-boot                     pass
>  5 debian-install               pass
>  6 debian-fixup                 pass
>  7 guest-start                  pass
>  8 guest-saverestore            pass
>  9 guest-localmigrate           pass
> 10 guest-stop                   fail
> 11 capture-logs(11)             pass
> 
> ------------------------------------------------------------
> sg-report-flight on woking.cam.xci-test.com
> logs: /home/xc_osstest/logs
> images: /home/xc_osstest/images
> 
> Logs, config files, etc. are available at
>     http://www.chiark.greenend.org.uk/~xensrcts/logs
> 
> Test harness code can be found at
>     http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
> 
> 
> Not pushing.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.