Re: [PATCH v5 1/3] automation: Create Yocto docker images

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Michal,

> On 30 Nov 2022, at 13:27, Michal Orzel <michal.orzel@xxxxxxx> wrote:
> 
> Hi Bertrand,
> 
> I will try to build and run the yocto containers for a test.
> In the meantime I just spotted one thing.

I will wait until tomorrow for a v6 in case you find issues.

> 
> On 30/11/2022 13:15, Bertrand Marquis wrote:
>> 
>> 
>> Add containers suitable to run yocto kirkstone build based on ubuntu
>> 22.04. It contains all packages required by Yocto and a checkout of the
>> layers required to build Xen with Yocto.
>> 
>> Add a generic docker image template to be used to automatically generate
>> docker files for different configurations:
>> - specific yocto version
>> - different targets (qemu arm, arm64 and x86)
>> - different host platforms (x86 or arm64)
>> 
>> During a call to 'make all', only the images for the host platform we
>> run it on in the CI is generated.
>> If needed, images for an other host platform can be generated manually
>> by calling the right make target (see make help).
>> 
>> Add a build script to build and run xen on qemu using Yocto.
>> The script supports arm32, arm64 and x86_64 and checks that dom0 is
>> properly booting. At this stage this does not run any guest on top of
>> dom0. The script is to be executed in one of the docker images to build
>> and run a system using a Xen source tree.
>> 
>> Signed-off-by: Bertrand Marquis <bertrand.marquis@xxxxxxx>
>> ---
>> Changes in v5:
>> - typo fixes
>> - add INTERMEDIATE to remove intermediate dockerfile
>> - use container architecture
>> - use full words in foreach loops
>> - fix shellcheck findings in build-yocto.sh
>> - rebase on sstabellini next branch
>> Changes in v4:
>> - Rework the system to have one dockerfile template from which make will
>> generate the required dockerfiles for the wanted configuration
>> - add support for different host architectures
>> - Merge the generation system into one single dockerfile
>> - Merge patches 1 and 2 in a single patch
>> - Introduce CONTAINERS_EXTRA to have extra containers not built by
>> default (for those not used by CI but useful to users)
>> Changes in v3:
>> - limit number of jobs in yocto by default to 8 and add --num-jobs
>> option to the script to set a custom number of jobs
>> - do not copy anymore the build-yocto.sh script inside the main image so
>> that the current one in the repository is used when running
>> Changes in v2:
>> - add a --dump-log command line option to build-yocto.sh script to dump
>> the logs if an error occurs.
>> Changes in v1:
>> - add --image command line argument to build-yocto.sh to allow building
>> something different than xen-image-minimal.
>> - modify dockerfile to have one layer per line and make it easier to add
>> other. I kept the for loop to keep the number of docker steps lower
>> - update commit message to warn that no guest are tested.
>> - fix build-yocto script to properly return with an error if expect
>> script ends up in timeout or EOF.
>> ---
>> automation/build/Makefile                  |  14 +-
>> automation/build/yocto/build-yocto.sh      | 351 +++++++++++++++++++++
>> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>> automation/build/yocto/yocto.inc           |  42 +++
>> 4 files changed, 519 insertions(+), 2 deletions(-)
>> create mode 100755 automation/build/yocto/build-yocto.sh
>> create mode 100644 automation/build/yocto/yocto.dockerfile.in
>> create mode 100644 automation/build/yocto/yocto.inc
>> 
>> diff --git a/automation/build/Makefile b/automation/build/Makefile
>> index a4b2b85178cf..72a5335baec1 100644
>> --- a/automation/build/Makefile
>> +++ b/automation/build/Makefile
>> @@ -1,13 +1,18 @@
>> 
>> # the base of where these containers will appear
>> REGISTRY := registry.gitlab.com/xen-project/xen
>> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
>> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard 
>> */*.dockerfile)))
>> +CONTAINERS_EXTRA =
>> DOCKER_CMD ?= docker
>> 
>> +include yocto/yocto.inc
> Because this appears before help, when calling make without any target like:
> make -C automation/build
> the previous behavior of printing help by default is no longer true (the 
> default target is
> taken from yocto.inc in this case). Instead we're getting:
> 
> make: Entering directory...
> rm yocto/kirkstone-qemuarm64.dockerfile
> make: Leaving directory...
> 
> This include can be easily moved after help to stick to the current behavior 
> and still
> achieve what we want (i.e. make will still first evaluate include and then 
> call the first
> target which is help, so CONTAINERS and CONTAINERS_EXTRA will contain correct 
> data).

Complete right.
I will fix that in v6.

Cheers
Bertrand

> 
>> +
>> help:
>>        @echo "Builds containers for building Xen based on different distros"
>>        @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>>        @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
>> +       @echo "Extra containers (not built using make all):"
>> +       @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>>        @echo "To push container builds, set the env var PUSH"
>> 
>> %: %.dockerfile ## Builds containers
>> @@ -16,5 +21,10 @@ help:
>>                $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>>        fi
>> 
>> -.PHONY: all
>> +.PHONY: all clean
>> all: $(CONTAINERS)
>> +
>> +# Remove generated dockerfiles for yocto
>> +clean:
>> +       rm -f yocto/*.dockerfiles
>> +
> 
> ~Michal