Re: [RFC PATCH] xen/design: Add design for EFI dom0less system start

[Luca Fancellu <luca.fancellu@xxxxxxx>]

> On 8 Sep 2021, at 02:09, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> 
> On Tue, 7 Sep 2021, Julien Grall wrote:
>> On 07/09/2021 12:51, Luca Fancellu wrote:
>>>> On 7 Sep 2021, at 10:35, Julien Grall <julien@xxxxxxx> wrote:
>>>> 
>>>> Hi Luca,
>>>> 
>>>> On 07/09/2021 07:52, Luca Fancellu wrote:
>>>>> Add a design describing a proposal to improve the EFI
>>>>> configuration file, adding keywords to describe domU
>>>>> guests and allowing to start a dom0less system.
>>>>> Signed-off-by: Luca Fancellu <luca.fancellu@xxxxxxx>
>>>>> ---
>>>>>  docs/designs/efi-arm-dom0less.md | 105 +++++++++++++++++++++++++++++++
>>>>>  1 file changed, 105 insertions(+)
>>>>>  create mode 100644 docs/designs/efi-arm-dom0less.md
>>>>> diff --git a/docs/designs/efi-arm-dom0less.md
>>>>> b/docs/designs/efi-arm-dom0less.md
>>>>> new file mode 100644
>>>>> index 0000000000..8d8fa2243f
>>>>> --- /dev/null
>>>>> +++ b/docs/designs/efi-arm-dom0less.md
>>>>> @@ -0,0 +1,105 @@
>>>>> +# Xen EFI configuration file
>>>>> +
>>>>> +The current configuration file used by Xen when it is started as an EFI
>>>>> +application is considering only the dom0 guest and doesn't have any
>>>>> +property to describe and load in memory domU guests.
>>>> 
>>>> From my understanding, the problem is less about properties (we already
>>>> have them in the Device-Tree) but more about where are the binaries
>>>> located in memory as we don't know in advance.
>>> 
>>> Hi Julien,
>> Hi Luca,
>> 
>>> I think I used the wrong word there, I meant “keyword” instead of “property”
>>> because I was referring about the
>>> lack of keywords to describe a domu guest in the Xen EFI configuration file.
>>> 
>>> I agree with you that on systems with static allocation, the kernel and
>>> ramdisk binaries must be at certain locations
>>> that are out of control when we use the EFI boot services, the thing we can
>>> do is provide a keyword to specify the
>>> addresses and then use the CopyMem() function to relocate the kernel/ramdisk
>>> in the address we want.
>> 
>> I wasn't specifically referring to static allocation here, sorry if this
>> wasn't clear. I was pointing out that most of the information you create in
>> the xen.cfg is going to be similar to what we already provide in the
>> Device-Tree.
>> 
>> My main concern is everytime we add a new feature in Dom0less, a developer
>> would need to write code for the DT and UEFI. This will increase the code 
>> size
>> and maintenance.
>> 
>> The same can be said for the admin as if they want to switch from plain 
>> U-boot
>> to UEFI, they would also need to fully rewrite the bindings.
>> 
>> AFAICT, most of the information provided in the Device-Tree are usable even
>> when using UEFI. So I would prefer if we try to re-use what's existing. This
>> is what my proposal below was about.
>> 
>>> 
>>>> 
>>>> So I would like to propose something that build on top of the Device-Tree
>>>> work we did. Note this is early thoughts.
>>>> 
>>>> The problematic nodes in the DT are:
>>>> 
>>>>        module@0x4a000000 {
>>>>            compatible = "multiboot,kernel", "multiboot,module";
>>>>            reg = <0x0 0x4a000000 0xffffff>;
>>>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>>>        };
>>>> 
>>>>        module@0x4b000000 {
>>>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>>>            reg = <0x0 0x4b000000 0xffffff>;
>>>>        };
>>>> 
>>>> In particular the property "reg" cannot be known in advance because the
>>>> UEFI stub will be responsible to load the binaries in memory.
>>> 
>>> Yes that’s true, the UEFI stub is using from the UEFI boot service the
>>> AllocatePages function that is giving back an address out of our control,
>>> then using another function the binary is read from the disk and copied at
>>> that address, finally the UEFI stub is writing the node in the device tree
>>> that
>>> will be used by Xen later.
>> 
>> I am not sure to follow. Are you saying the UEFI stub will create the 
>> dom0less
>> node in the DT based on the xen.cfg?
>> 
>>> 
>>>> 
>>>> What we could do is providing a list of binaries to load and associate a
>>>> key for each of them. Something like:
>>>> 
>>>> binary=<binary> <key>
>>>> binary=<binary2> <key2>
>>>> ....
>>>> 
>>>> We can then replace the property "reg" with a new property "uefi,key" that
>>>> will contain the name of the binary.
>>>> 
>>>> What do you think?
>>> 
>>> Here I’m lost, because I don’t understand what we are going to do with the
>>> name of the binary.
>> 
>> <binaryX> would be used by the UEFI stub to load the binary in memory. Each
>> binary will have a <keyX> which helps to refer them in the Device-Tree. To
>> give a concrete example, let say we have two dom0less domains:
>>  - DomA: 2 vCPUs, 128MB
>>  - DomB: 3 vCPUs, 512MB
>> 
>> DomA and DomB will be using the same kernel but a different ramdisk. xen.cfg,
>> would look like:
>> 
>> [global]
>> default=section1
>> 
>> [section1]
>> options=console=vga,com1 com1=57600 loglvl=all noreboot
>> kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>> ramdisk=initrd-3.0.31-0.4-xen
>> xsm=<filename>
>> dtb=devtree.dtb
>> binary=vmlinuz-guest domu-kernel
>> binary=ramdisk-domA.img domA-ramdisk
>> binary=ramdisk-domB.img domB-ramdisk
>> 
>> The chosen node in the DT would look like:
>> 
>> chosen {
>>    domU1 {
>>        compatible = "xen,domain";
>>        #address-cells = <0x2>;
>>        #size-cells = <0x1>;
>>        memory = <0 0x8000000>;
>>        cpus = <2>;
>> 
>>        module@1 {
>>            compatible = "multiboot,kernel", "multiboot,module";
>>            uefi,binary = "domu-kernel";
>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>        };
>> 
>>        module@2 {
>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>            uefi,binary = "domA-ramdisk";
>>        };
>>    };
>> 
>>    domU2 {
>>        compatible = "xen,domain";
>>        #address-cells = <0x3>;
>>        #size-cells = <0x1>;
>>        memory = <0 0x20000000>;
>>        cpus = <3>;
>> 
>>        module@1 {
>>            compatible = "multiboot,kernel", "multiboot,module";
>>            uefi,binary = "domu-kernel";
>>            bootargs = "console=ttyAMA0 init=/bin/sh";
>>        };
>> 
>>        module@2 {
>>            compatible = "multiboot,ramdisk", "multiboot,module";
>>            uefi,binary = "domA-ramdisk";
>>        };
>>    };
>> };
>> 
>> With this approach, the change is quite minimal to move between an classic
>> U-boot boot and EFI boot.
> 
> Great idea! I think it is good to try to reuse Device Tree, and using it
> as configuration is well aligned with other projects too (e.g.  Zephyr).
> 
> 
> There are a few options for the bindings. These are some ideas.
> 
> If we are just going to specify a filename and a reference with the
> "binary" key=value pair, then we could get rid of it entirely and just
> write the filename directly in device tree:
> 
>    domU1 {
>        compatible = "xen,domain";
>        #address-cells = <0x2>;
>        #size-cells = <0x1>;
>        memory = <0 0x8000000>;
>        cpus = <2>;
> 
>        module@1 {
>            compatible = "multiboot,kernel", "multiboot,module";
>            uefi,binary = "vmlinuz-guest";
>            bootargs = "console=ttyAMA0 init=/bin/sh";
>        };
>    };
> 
> Then we don't even need the binary key=value pair, and especially the
> device tree becomes "self contained" as in we don't need to resolve the
> reference to "binary=value ref" to understand it.
> 
> 
> (Also worth noting as inspiration the HyperLaunch binding that is using
> the "mb-index" property as index in the multiboot module chain. Not
> directly applicable, but interesting.)

Hi Stefano,

Yes this is an amazing idea, it will solve every issue without introducing new 
keywords in the config file
and it maintains the compatibility.
Now in the current code there is a check for modules that are 
“multiboot,module”, if any is found the configuration
file is skipped, but if we change this bit to check only for modules whose 
parent is the chosen node, we can
distinguish very well between the dom0 and domus kernel.

If we all agree with this, I will update the design.

Cheers,

Luca