[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] [PATCH osstest v5] Add libvirt build and test jobs.



I have run this through a "cr-daily-branch --real" and the push gate machinery
seem functional. The test cases run through to starting a guest which fails
because virsh support has not been implemented yet.

The libvirt.git initscript is horribly redhat specific, and the debian package
one is large and complex, so ts-libvirt-build inserts its own simple
initscript.

I have observed the autogen.sh phase taking 34 minutes on army (arm build
cluster) hence the surprisingly large timeout of an hour. The build took around
20 minutes on the same system, but I used a timeout of an hour anyway to be
safe.

The standalone wrapper is updated to handle path_libvirt in set-paths.

libvirt.git's autogen.sh involves initialising/cloning a git submodule, which
doesn't use osstest's git cache and introduces unpredicability into what is
built. Therefore ap-* are taught about a new libvirt-gnulib tree (prefixed
libvirt incase we end up with other users of gnulib in the future). However
this doesn't use the usual push gate mechanism, since this was deemed to be
overkill and just always uses the tip of gnulib. The results are recorded
though so the bisector can still operate over this tree in the event of
breakage.

Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
Acked-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
Cc: Jim Fehlig <jfehlig@xxxxxxxx>
Cc: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
---
v5: Print message in ap-push when not pushing gnulib
    Switch from libvirt-gnulib to gnulib-libvirt in all tree names. Implemented
    with:
        sed -i -e 's/\(libvirt\)\([-_]\)\(gnulib\)/\3\2\1/gi' $(git grep -il 
libvirt.gnulib)
    And a manual inspection of the result.
v4: Use "xen-tested-master" as the push gate output.
    Install extra libraries needed by the binary onto the target.
    Install build dependencies in ts-xen-build-prep instead of
    ts-libvirt-build. Although apt-get invocations are now locked I still have
    concerns:
    http://lists.xen.org/archives/html/xen-devel/2014-04/msg01951.html
    Clone gnulib separately.
    Avoid running stuff as root during build, no need
    Correct ap-fetch-version-old (actually use the gate, not the main libvirt 
tree)
    Increase timeouts for arm build.
    Ran through cr-daily-branch
v3: Disable uml, lxc and vbox backends at build time
    Create test job on armhf as well as x86
    Don't run non-libvirt tests in libvirt flights.
v2: Add a test job, which will clearly fail.
    Make ts-xen-install DTRT though
---
 Osstest/TestSupport.pm |   8 ++-
 ap-common              |  10 +++
 ap-fetch-version       |   8 +++
 ap-fetch-version-old   |   9 +++
 ap-print-url           |   6 ++
 ap-push                |   9 +++
 cr-daily-branch        |  12 ++++
 cri-common             |   1 +
 make-flight            |   7 +++
 mfi-common             |  17 +++++-
 sg-run-job             |   5 ++
 standalone             |   2 +-
 ts-libvirt-build       | 162 +++++++++++++++++++++++++++++++++++++++++++++++++
 ts-xen-build-prep      |   3 +
 ts-xen-install         |   7 ++-
 15 files changed, 262 insertions(+), 4 deletions(-)
 create mode 100755 ts-libvirt-build

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index d49ff4f..4ea59a6 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1774,7 +1774,13 @@ our %toolstacks=
         Command => 'xl',
         CfgPathVar => 'cfgpath',
        RestoreNeedsConfig => 1,
-        }
+        },
+     'libvirt' => {
+        NewDaemons => [qw(libvirtd)],
+        Dom0MemFixed => 1,
+        Command => 'virsh',
+        ExtraPackages => [qw(libnl1 libavahi-client3)],
+        },
      );
 
 sub toolstack () {
diff --git a/ap-common b/ap-common
index cb0cfff..173f4be 100644
--- a/ap-common
+++ b/ap-common
@@ -33,6 +33,14 @@
 
 : ${TREEVCS_LINUX:=git}
 
+: ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git}
+: ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git}
+: ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git}
+: ${TREEVCS_LIBVIRT:=git}
+
+: ${TREE_GNULIB_LIBVIRT:=git://git.sv.gnu.org/gnulib.git}
+: ${TREEVCS_GNULIB_LIBVIRT:=git}
+
 : ${TREE_LINUXFIRMWARE:=git://xenbits.xen.org/osstest/linux-firmware.git}
 : ${PUSH_TREE_LINUXFIRMWARE:=$XENBITS:/home/osstest/ext/linux-firmware.git}
 : 
${UPSTREAM_TREE_LINUXFIRMWARE:=$GIT_KERNEL_ORG/pub/scm/linux/kernel/git/firmware/linux-firmware.git}
@@ -56,6 +64,8 @@ fi
 
 : ${LOCALREV_XEN:=daily-cron.$branch}
 : ${LOCALREV_LINUX:=daily-cron.$branch}
+: ${LOCALREV_LIBVIRT:=daily-cron.$branch}
+: ${LOCALREV_GNULIB_LIBVIRT:=daily-cron.$branch}
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index aaf884d..7cc2d98 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -69,6 +69,14 @@ linuxfirmware)
        repo_tree_rev_fetch_git linuxfirmware \
                $UPSTREAM_TREE_LINUXFIRMWARE master daily-cron.$branch
        ;;
+libvirt)
+       repo_tree_rev_fetch_git libvirt \
+               $TREE_LIBVIRT master $LOCALREV_LIBVIRT
+       ;;
+gnulib-libvirt)
+       repo_tree_rev_fetch_git gnulib-libvirt \
+               $TREE_GNULIB_LIBVIRT master $LOCALREV_GNULIB_LIBVIRT
+       ;;
 osstest)
        if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
            git fetch $HOME/testing.git pretest:ap-fetch >&2
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 21ff786..3879e09 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -27,6 +27,7 @@ select_xenbranch
 
 : ${BASE_TAG_LINUX2639:=tested/2.6.39.x}
 : ${BASE_LOCALREV_LINUX:=daily-cron.$branch.old}
+: ${BASE_LOCALREV_LIBVIRT:=daily-cron.$branch.old}
 
 : ${BASE_TREE_QEMU_UPSTREAM:=${TREE_QEMU_UPSTREAM/\/staging\//\/}}
 
@@ -73,6 +74,14 @@ linuxfirmware)
        repo_tree_rev_fetch_git linuxfirmware \
                $TREE_LINUXFIRMWARE master daily-cron-old.$branch
        ;;
+libvirt)
+       repo_tree_rev_fetch_git libvirt \
+               $BASE_TREE_LIBVIRT xen-tested-master $BASE_LOCALREV_LIBVIRT
+       ;;
+gnulib-libvirt)
+       # No push gate, same as ap-fetch-version
+       ./ap-fetch-version $branch
+       ;;
 osstest)
        if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
            git fetch -f $HOME/testing.git incoming:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index f2812d0..b1af0b9 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -46,6 +46,12 @@ linux-2.6.39)
 linuxfirmware)
        echo $UPSTREAM_TREE_LINUXFIRMWARE
        ;;
+libvirt)
+       echo $TREE_LIBVIRT
+       ;;
+gnulib-libvirt)
+       echo $TREE_GNULIB_LIBVIRT
+       ;;
 osstest)
        echo none:;
        ;;
diff --git a/ap-push b/ap-push
index dc0a166..69f88a3 100755
--- a/ap-push
+++ b/ap-push
@@ -32,6 +32,7 @@ select_xenbranch
 TREE_LINUX=$PUSH_TREE_LINUX
 TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
 TREE_XEN=$PUSH_TREE_XEN
+TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
 
 if info_linux_tree "$branch"; then
        cd $repos/linux
@@ -72,6 +73,14 @@ linux-xen-*)
         cd $repos/linux
         git push $TREE_LINUX $revision:refs/heads/$branch
         ;;
+libvirt)
+       cd $repos/libvirt
+       git push $TREE_LIBVIRT $revision:xen-tested-master
+       ;;
+gnulib-libvirt)
+       # No gate
+       echo "gnulib-libvirt has not push gate, refusing to push" >&2
+       exit 1
 osstest)
        git push $HOME/testing.git $revision:incoming
        git push $XENBITS:/home/xen/git/osstest.git $revision:master
diff --git a/cr-daily-branch b/cr-daily-branch
index 02fef15..05fdee1 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -133,6 +133,14 @@ if [ "x$REVISION_LINUX_ARM" = x ]; then
         export REVISION_LINUX_ARM
     fi
 fi
+if [ "x$REVISION_LIBVIRT" = x ]; then
+       determine_version REVISION_LIBVIRT libvirt
+       export REVISION_LIBVIRT
+fi
+if [ "x$REVISION_GNULIB_LIBVIRT" = x ]; then
+       determine_version REVISION_GNULIB_LIBVIRT gnulib-libvirt
+       export REVISION_GNULIB_LIBVIRT
+fi
 if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
        determine_version REVISION_LINUXFIRMWARE linuxfirmware
         export REVISION_LINUXFIRMWARE
@@ -165,6 +173,10 @@ linuxfirmware)
        realtree=linuxfirmware
        NEW_REVISION=$REVISION_LINUXFIRMWARE
         ;;
+libvirt)
+       realtree=libvirt
+       NEW_REVISION=$REVISION_LIBVIRT
+       ;;
 *)
        NEW_REVISION=''
        wantpush=false
diff --git a/cri-common b/cri-common
index 497d4e3..f472927 100644
--- a/cri-common
+++ b/cri-common
@@ -42,6 +42,7 @@ select_xenbranch () {
        linux)                  tree=linux;     xenbranch=xen-unstable ;;
        linux-*)                tree=linux;     xenbranch=xen-unstable ;;
        linuxfirmware)      tree=linuxfirmware; xenbranch=xen-unstable ;;
+       libvirt)                tree=libvirt;   xenbranch=xen-unstable ;;
        osstest)                tree=osstest;   xenbranch=xen-unstable ;;
        esac
        if [ "x$tree" = xlinux ]; then
diff --git a/make-flight b/make-flight
index 92b813a..aadf58b 100755
--- a/make-flight
+++ b/make-flight
@@ -64,6 +64,9 @@ job_create_test_filter_callback () {
           return 1;;
       esac
       ;;
+    libvirt)
+      if [ x$toolstack != xlibvirt ] ; then return 1; fi
+      ;;
     *)
       case "$job" in
         *-qemuu-*)
@@ -238,6 +241,10 @@ test_matrix_do_one () {
             $xenarch $dom0arch                                   \
             $debian_runvars all_hostflags=$most_hostflags
 
+  job_create_test test-$xenarch$kern-$dom0arch-libvirt test-debian libvirt \
+            $xenarch $dom0arch                                       \
+            $debian_runvars all_hostflags=$most_hostflags
+
   # No further arm tests at the moment
   if [ $dom0arch = armhf ]; then
       return
diff --git a/mfi-common b/mfi-common
index 598aa02..235575e 100644
--- a/mfi-common
+++ b/mfi-common
@@ -162,6 +162,18 @@ create_build_jobs () {
                 tree_linuxfirmware=$TREE_LINUXFIRMWARE                       \
                 revision_linuxfirmware=$REVISION_LINUXFIRMWARE
 
+    ./cs-job-create $flight build-$arch-libvirt build-libvirt                \
+                arch=$arch                                                   \
+        tree_xen=$TREE_XEN                                                   \
+                $RUNVARS $BUILD_RUNVARS $BUILD_LIBVIRT_RUNVARS $arch_runvars \
+                $suite_runvars                                               \
+                host_hostflags=$build_hostflags                              \
+                buildjob=${bfi}build-$arch                                   \
+                tree_libvirt=$TREE_LIBVIRT revision_libvirt=$REVISION_LIBVIRT\
+                ${TREEVCS_LIBVIRT:+treevcs_libvirt=}${TREEVCS_LIBVIRT}       \
+                tree_gnulib_libvirt=$TREE_GNULIB_LIBVIRT 
revision_gnulib_libvirt=$REVISION_GNULIB_LIBVIRT\
+                
${TREEVCS_GNULIB_LIBVIRT:+treevcs_gnulib_libvirt=}${TREEVCS_GNULIB_LIBVIRT}
+
     case "$arch" in
     armhf) continue;; # don't do any other kernel builds
     esac
@@ -193,6 +205,7 @@ job_create_test () {
 
   xenbuildjob="${bfi}build-$xenarch"
   buildjob="${bfi}build-$dom0arch"
+  tsbuildjob=
 
   case "$xenbranch:$toolstack" in
     xen-3.*-testing:*) ;;
@@ -203,11 +216,13 @@ job_create_test () {
     *:xend) xenbuildjob="$xenbuildjob-xend"
             buildjob="${bfi}build-$dom0arch-xend"
             ;;
+    *:libvirt) tsbuildjob="libvirtbuildjob=$buildjob-libvirt"
+            ;;
   esac
 
   ./cs-job-create $flight $job $recipe toolstack=$toolstack       \
     $RUNVARS $TEST_RUNVARS $most_runvars                          \
-    xenbuildjob=$xenbuildjob buildjob=$buildjob "$@"
+    xenbuildjob=$xenbuildjob buildjob=$buildjob $tsbuildjob "$@"
 }
 
 # Iterate over xenarch, dom0arch and kernel calling test_matrix_do_one
diff --git a/sg-run-job b/sg-run-job
index d336fbd..3d44331 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -312,6 +312,7 @@ proc test-guest-nomigr {g} {
 
 proc need-hosts/build {} { return BUILD }
 proc need-hosts/build-kern {} { return BUILD }
+proc need-hosts/build-libvirt {} { return BUILD }
 
 proc run-job/build {} {
     run-ts . = ts-xen-build
@@ -321,6 +322,10 @@ proc run-job/build-kern {} {
     run-ts . = ts-kernel-build
 }
 
+proc run-job/build-libvirt {} {
+    run-ts . = ts-libvirt-build
+}
+
 proc prepare-build-host {} {
     global jobinfo
     run-ts broken = ts-hosts-allocate + host
diff --git a/standalone b/standalone
index 8d9d88a..6b08995 100755
--- a/standalone
+++ b/standalone
@@ -171,7 +171,7 @@ case $op in
 
        job=$1; shift
 
-       for d in '' xen kern ; do
+       for d in '' xen kern libvirt ; do
            runvar="path_${d}dist"
            path="build/${d}dist.tar.gz"
            if [ -f "logs/$flight/$job/$path" ] ; then
diff --git a/ts-libvirt-build b/ts-libvirt-build
new file mode 100755
index 0000000..9c731ce
--- /dev/null
+++ b/ts-libvirt-build
@@ -0,0 +1,162 @@
+#!/usr/bin/perl
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+my $leaf= "build.$flight.$job";
+my $homedir = get_host_property($ho, 'homedir', '/home/osstest');
+my $builddir= "$homedir/$leaf";
+my $xendist= "$builddir/xendist";
+
+sub libvirtd_init ();
+
+my $makeflags= get_host_property($ho, 'build make flags', '-j4');
+
+sub checkout () {
+    target_cmd($ho, "rm -rf $builddir && mkdir $builddir && mkdir $xendist", 
600);
+
+    my $path = get_stashed("path_dist", $r{"buildjob"});
+    my $distcopy= "$builddir/dist.tar.gz";
+    target_putfile($ho, 300, $path, $distcopy);
+    target_cmd($ho, "tar -C $xendist -hzxf $distcopy", 300);
+
+    build_clone($ho, 'libvirt', $builddir, 'libvirt');
+    build_clone($ho, 'gnulib_libvirt', $builddir, 'gnulib-libvirt');
+}
+
+sub config() {
+    # Uses $GNULIB_SRCDIR because ./autogen.sh doesn't propagate
+    # --gnulib-srcdir to ./bootstap.
+    target_cmd_build($ho, 3600, $builddir, <<END);
+        cd libvirt
+        CFLAGS="-I$xendist/usr/local/include/" \\
+        LDFLAGS="-L$xendist/usr/local/lib/ 
-Wl,-rpath-link=$xendist/usr/local/lib/" \\
+       GNULIB_SRCDIR=$builddir/gnulib-libvirt \\
+            ./autogen.sh --no-git \\
+                         --with-libxl --without-xen --without-xenapi 
--without-selinux \\
+                         --without-lxc --without-vbox --without-uml \\
+                         --sysconfdir=/etc --localstatedir=/var #/
+END
+}
+
+sub build() {
+    target_cmd_build($ho, 3600, $builddir, <<END);
+        cd libvirt
+        (make $makeflags 2>&1 && touch ../build-ok-stamp) |tee ../log
+        test -f ../build-ok-stamp #/
+        echo ok.
+END
+}
+
+sub install() {
+    target_cmd_build($ho, 300, $builddir, <<END);
+        mkdir -p dist
+        cd libvirt
+        make $makeflags install DESTDIR=$builddir/dist
+        mkdir -p $builddir/dist/etc/init.d
+END
+    target_putfilecontents_stash($ho, 60,
+                                 libvirtd_init(),
+                                 $builddir.'/dist/etc/init.d/libvirtd');
+    target_cmd_build($ho, 60, $builddir, <<END);
+        chmod +x $builddir/dist/etc/init.d/libvirtd
+END
+}
+
+sub collectversion_gnulib () {
+    store_revision($ho, 'gnulib_libvirt', "$builddir/gnulib-libvirt", 1);
+}
+
+checkout();
+config();
+build();
+install();
+collectversion_gnulib();
+built_stash($ho, $builddir, 'dist', 'libvirtdist');
+
+sub libvirtd_init () {
+    return <<'END';
+#! /bin/sh
+#
+# Init script for libvirtd
+#
+# Loosely based on Debian libvirt-bin initscript:
+# (c) 2007 Guido Guenther <agx@xxxxxxxxxxx>
+# based on the skeletons that comes with dh_make
+#
+### BEGIN INIT INFO
+# Provides:          libvirt-bin libvirtd
+# Required-Start:    $network $local_fs $remote_fs $syslog
+# Required-Stop:     $local_fs $remote_fs $syslog
+# Should-Start:      avahi-daemon cgconfig
+# Should-Stop:       avahi-daemon cgconfig
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: libvirt management daemon
+### END INIT INFO
+
+PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+export PATH
+DAEMON=/usr/local/sbin/libvirtd
+NAME=libvirtd
+DESC="libvirt management daemon"
+
+test -x $DAEMON || exit 0
+. /lib/lsb/init-functions
+
+PIDFILE=/var/run/$NAME.pid
+
+case "$1" in
+  start)
+       [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
+       start-stop-daemon --start --pidfile $PIDFILE \
+           --exec $DAEMON -- -d $libvirtd_opts
+       [ "$VERBOSE" != no ] && log_end_msg $?
+       ;;
+  stop)
+       [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
+       start-stop-daemon --stop --pidfile $PIDFILE \
+           --exec $DAEMON
+       [ "$VERBOSE" != no ] && log_end_msg $?
+       ;;
+  restart)
+       stop
+       start
+       ;;
+  reload|force-reload)
+       start-stop-daemon --stop --signal 1 --quiet --pidfile \
+            /var/run/$NAME.pid --exec $DAEMON
+       ;;
+  *)
+       N=/etc/init.d/libvirtd
+       echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
+       exit 1
+       ;;
+esac
+
+exit 0
+END
+}
diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 528d0a4..a4e166e 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -172,6 +172,9 @@ sub prep () {
                                libx11-dev git-core uuid-dev gettext gawk
                                libsdl-dev libyajl-dev libaio-dev 
libpixman-1-dev
                                libglib2.0-dev pkg-config
+                               autoconf automake libtool xsltproc
+                               libxml2-utils libxml2-dev libnl-dev
+                               libdevmapper-dev w3c-dtd-xhtml
                               ccache));
 
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
diff --git a/ts-xen-install b/ts-xen-install
index 4a810e4..ad0254c 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -60,10 +60,15 @@ sub packages () {
     if ($r{arch} eq 'i386') {
        target_install_packages($ho, 'libc6-xen');
     }
+    target_install_packages($ho, @{toolstack()->{ExtraPackages}})
+        if toolstack()->{ExtraPackages};
 }
 
 sub extract () {
-    foreach my $part ('', 'kern', 'xen') {
+    my @parts = ('', 'kern', 'xen');
+    push @parts, 'libvirt' if $r{toolstack} eq "libvirt";
+
+    foreach my $part (@parts) {
         target_extract_jobdistpath($ho, $part, "path_${part}dist",
                                   $r{"${part}buildjob"}, \%distpath);
     }
-- 
1.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.