Re: [PATCH v2] releases: use newer compression methods for tarballs

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 02/09/2025 5:15 pm, Andrew Cooper wrote:
> On 25/08/2025 2:54 pm, Jan Beulich wrote:
>> Other projects have long switched to xz and/or lzip.
>>
>> Tidy things some as well: With the removal of qemu from the tarball,
>> intermediately extracting the tarball again has become wasteful. Drop
>> that. Invoke compressors using asynchronous lists, to reduce overall
>> latency. Drop the -v option from the (previously implicit) gzip
>> invocation.
>>
>> Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
>> ---
>> v2: Don't expand intermediate uncompressed tarball. Don't check for
>>     commands' availablity. Don't request statistics. Use async lists.
>>
>> --- a/docs/process/release-technician-checklist.txt
>> +++ b/docs/process/release-technician-checklist.txt
>> @@ -119,7 +119,7 @@ RELEASE TARBALL
>>         make src-tarball           # uses git-describe (best for RCs)
>>          # ^find some way to add git-cache-proxy to this (done in 
>> ~iwj/.gitconfig)
>>         mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
>> -       mv dist/xen-$v.tar.gz 
>> /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
>> +       mv dist/xen-$v.tar.[glx]z 
>> /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
>>  
>>         # website-thing/xen.org is cvs -d 
>> mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
>>      cd /volatile/iwj/website-thing/xen.org
>> @@ -139,9 +139,12 @@ RELEASE TARBALL
>>      cvs add -kb oss-xen/release/$v/
>>  
>>          cd oss-xen/release/$v
>> -        gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
>> -    cvs add -kb xen-$v.tar.gz
>> -        cvs add -kb xen-$v.tar.gz.sig
>> +        for t in xen-$v.tar.[glx]z
>> +        do
>> +            gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
>> +            cvs add -kb $t
>> +            cvs add -kb $t.sig
>> +        done
>>          cd ../../..
>>  
>>      cvs ci -m $v
>> @@ -152,6 +155,10 @@ RELEASE TARBALL
>>      # should show something like
>>      #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
>>      #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
>> +    #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
>> +    #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
>> +    #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
>> +    #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
>>  
>>  After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
>>  
>> --- a/docs/process/xen-release-management.pandoc
>> +++ b/docs/process/xen-release-management.pandoc
>> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
>>  https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
>>  
>>  For your convenience there is also a tarball at:
>> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
>> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
>>  
>>  And the signature is at:
>> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
>> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
>>  
>>  Please send bug reports and test reports to xen-devel@xxxxxxxxxxxxxxxxxxxx.
>>  When sending bug reports, please CC relevant maintainers and me
>> --- a/tools/misc/mktarball
>> +++ b/tools/misc/mktarball
>> @@ -5,14 +5,6 @@
>>  # Takes 2 arguments, the path to the dist directory and the version
>>  set -ex
>>  
>> -function git_archive_into {
>> -    mkdir -p "$2"
>> -
>> -    git --git-dir="$1"/.git \
>> -    archive --format=tar HEAD | \
>> -    tar Cxf "$2" -
>> -}
>> -
>>  if [[ -z "$1" || -z "$2" ]] ; then
>>    echo "usage: $0 path-to-XEN_ROOT xen-version"
>>    exit 1
>> @@ -21,14 +13,20 @@ fi
>>  xen_root="$1"
>>  desc="$2"
>>  
>> -tdir="$xen_root/dist/tmp.src-tarball"
>> +tdir="$xen_root/dist"
>>  
>> -rm -rf $tdir
>> +rm -f $tdir/xen-$desc.tar.[glx]z
> This is asymmetric with the rm at the end.  I'd remove
> $tdir/xen-$desc.tar* here and remove the final rm.
>
> Looking at the uncompressed tarball is part of my process, and it was
> preserved previously.
>
> With something along these lines, Reviewed-by: Andrew Cooper
> <andrew.cooper3@xxxxxxxxxx>
>
>>  
>>  mkdir -p $tdir
>>  
>> -git_archive_into $xen_root $tdir/xen-$desc
>> +git --git-dir="$xen_root/.git" archive --format=tar HEAD 
>> --prefix=xen-$desc/ \
>> +    >"$tdir/xen-$desc.tar"
>> +
>> +gzip -9k "$tdir/xen-$desc.tar" &
>> +xz -9k "$tdir/xen-$desc.tar" &
>> +lzip -9k "$tdir/xen-$desc.tar" &
>> +wait
> Interestingly, this wasn't fatal for not having lzip, but the error was
> clear on the console given the reduced verbosity, and doing 3 at the
> same time worked very nicely.
>
>>  
>> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>> +rm -f $tdir/xen-$desc.tar
>>  
>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z
> This was grammatically awkward to begin with, but is now pretty useless,
> especially combined with the set -x so it gets printed twice.
>
> Something like this:
>
> echo "Source tarballs:"
> ls -lah $tdir/xen-$desc.tar*
>
> generates:
>
> -rw-rw-r-- 1 andrew andrew  32M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar
> -rw-rw-r-- 1 andrew andrew 6.8M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
>
>
> on my system and is rather more useful IMO.

And just for reference, here is what zstd looks like:

-rw-rw-r-- 1 andrew andrew  32M Sep  2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar
-rw-rw-r-- 1 andrew andrew 6.8M Sep  2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
-rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
-rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
-rw-rw-r-- 1 andrew andrew 4.9M Sep  2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.zst

using -19kq  (zstd defaults to giving an interactive progress bar, and
needs quietening).

With a bit of `time` hacking, gzip is 2.4s, zx is 11.8s, zstd is 13s and
lzip is 17.3s on my system.

~Andrew