[Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs

[Anthony PERARD <anthony.perard@xxxxxxxxxx>]

GitLab have a caching capability, see [1]. Let's use it to avoid using
Internet too often.

The cache is setup so that when xen.git/Config.mk is changed, the
cache will need to be recreated. This has been chosen because that is
where the information about how to clone sub-project trees is encoded
(revisions). That may not work for qemu-xen tree which usually is
`master', but that should be fine for now.

The cache is populated of "git bundle" which will contain a mirror of
the original repo, and can be cloned from. If the bundle exist, the
script have the Xen makefiles clone from it, otherwise it will clone
from the original URL and the bundles will be created just after.

We have more than one runner in GitLab, and no shared cache between
them, so every build jobs will be responsible to create the cache.

[1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache

Signed-off-by: Anthony PERARD <anthony.perard@xxxxxxxxxx>
---
 automation/gitlab-ci/build.yaml     |  8 +++++
 automation/scripts/prepare-cache.sh | 52 +++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)
 create mode 100755 automation/scripts/prepare-cache.sh

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 1e61d30c8545..8f9f53a4222f 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -1,6 +1,14 @@
 .build-tmpl: &build
   stage: build
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  cache:
+    key:
+      files:
+        - Config.mk
+    paths:
+      - ci_cache
+  before_script:
+    - ./automation/scripts/prepare-cache.sh
   script:
     - ./automation/scripts/build 2>&1 | tee build.log
   artifacts:
diff --git a/automation/scripts/prepare-cache.sh 
b/automation/scripts/prepare-cache.sh
new file mode 100755
index 000000000000..017f1b8f0672
--- /dev/null
+++ b/automation/scripts/prepare-cache.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+
+set -ex
+
+cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
+mkdir -p "$cachedir"
+
+declare -A r
+r[extras/mini-os]=MINIOS_UPSTREAM_URL
+r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
+r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
+r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
+r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
+
+bundle_loc() {
+    echo "$cachedir/${1//\//_}.git.bundle"
+}
+for d in ${!r[@]}; do
+    if [ -e $(bundle_loc $d) ]; then
+        export ${r[$d]}=$(bundle_loc $d)
+    fi
+done
+
+if ! make subtree-force-update-all; then
+    # There's maybe an issue with one of the git bundle, just clear the cache
+    # and allow it to be rebuilt by a different jobs.
+    # Make will reclone missing clones from original URLs instead of from the
+    # bundle.
+    for d in ${!r[@]}; do
+        rm -f "$(bundle_loc $d)"
+    done
+    exit
+fi
+
+
+tmpdir=$(mktemp -d "$CI_PROJECT_DIR/ci-tmp.XXX")
+for d in ${!r[@]}; do
+    bundle=$(bundle_loc $d)
+    if [ -e $bundle ]; then
+        # We didn't download anything new
+        continue
+    fi
+    # We create a mirror to be able to create a bundle that is a mirror of
+    # upstream. Otherwise, the bundle may not have refs that the build system
+    # will want, i.e. refs/heads/master would be missing from the bundle.
+    url=$(git --git-dir=$d/.git config remote.origin.url)
+    repo_mirrored="$tmpdir/${d//\//_}"
+    git clone --bare --mirror --reference "$d" "$url" "$repo_mirrored"
+    git --git-dir="$repo_mirrored" bundle create $bundle --all
+    rm -rf "$repo_mirrored"
+done
+rmdir "$tmpdir"
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel