Re: Proposal for Porting Xen to Armv8-R64 - DraftB

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Stefano,

Reply for non-Eventchannel comments:

On 2022/4/15 8:41, Stefano Stabellini wrote:
> On Fri, 25 Mar 2022, Wei Chen wrote:
> > # Proposal for Porting Xen to Armv8-R64
...
> > ## 2. Proposed changes of Xen
> > ### **2.1. Changes of build system:**
> >
> > - ***Introduce new Kconfig options for Armv8-R64***:
> >    Unlike Armv8-A, because lack of MMU support on Armv8-R64, we may not
> >    expect one Xen binary to run on all machines. Xen images are not common
> >    across Armv8-R64 platforms. Xen must be re-built for different Armv8-R64
> >    platforms. Because these platforms may have different memory layout and
> >    link address.
> >      - `ARM64_V8R`:
> >        This option enables Armv8-R profile for Arm64. Enabling this option
> >        results in selecting MPU. This Kconfig option is used to gate some
> >        Armv8-R64 specific code except MPU code, like some code for Armv8-R64
> >        only system ID registers access.
> >
> >      - `ARM_MPU`
> >        This option enables MPU on Armv8-R architecture. Enabling this option
> >        results in disabling MMU. This Kconfig option is used to gate some
> >        ARM_MPU specific code. Once when this Kconfig option has been enabled,
> >        the MMU relate code will not be built for Armv8-R64. The reason why
> >        not depends on runtime detection to select MMU or MPU is that, we don't
> >        think we can use one image for both Armv8-R64 and Armv8-A64. Another
> >        reason that we separate MPU and V8R in provision to allow to support MPU
> >        on 32bit Arm one day.
> >
> >    ***Try to use `if ( IS_ENABLED(CONFIG_ARMXXXX) )` instead of spreading***
> >    ***`#ifdef CONFIG_ARMXXXX` everywhere, if it is possible.***
> >
> > - ***About Xen start address for Armv8-R64***:
> >    On Armv8-A, Xen has a fixed virtual start address (link address too) on all
> >    Armv8-A platforms. In an MMU based system, Xen can map its loaded address
> >    to this virtual start address. On Armv8-A platforms, the Xen start address
> >    does not need to be configurable. But on Armv8-R platforms, they don't have
> >    MMU to map loaded address to a fixed virtual address. And different platforms
> >    will have very different address space layout, so it's impossible for Xen to
> >    specify a fixed physical address for all Armv8-R platforms' start address.
> >
> >    - `XEN_START_ADDRESS`
> >      This option allows to set the custom address at which Xen will be
> >      linked. This address must be aligned to a page size. Xen's run-time
> >      addresses are the same as the link time addresses.
> >      ***Notes: Fixed link address means the Xen binary could not be***
> >      ***relocated by EFI loader. So in current stage, Xen could not***
> >      ***be launched as an EFI application on Armv8-R64.(TODO#3.3)***
> >
> >      - Provided by platform files.
> >        We can reuse the existed arm/platforms store platform specific files.
> >        And `XEN_START_ADDRESS` is one kind of platform specific information.
> >        So we can use platform file to define default `XEN_START_ADDRESS` for
> >        each platform.
> >
> >      - Provided by Kconfig.
> >        This option can be an independent or a supplymental option. Users can
> >        define a customized `XEN_START_ADDRESS` to override the default value
> >        in platform's file.
> >
> >      - Generated from device tree by build scripts (optional)
> >        Vendors who want to enable Xen on their Armv8-R platforms, they can
> >        use some tools/scripts to parse their boards device tree to generate
> >        the basic platform information. These tools/scripts do not necessarily
> >        need to be integrated in Xen, but Xen can give some recommended
> >        configuration. For example, Xen can recommend Armv8-R platforms to use
> >        lowest ram start address + 2MB as the default Xen start address.
> >        The generated platform files can be placed to arm/platforms for
> >        maintenance.
> >
> >      - Enable Xen PIC/PIE (optional)
> >        We have mentioned about PIC/PIE in section 1.2. With PIC/PIE support,
> >        Xen can run from everywhere it has been loaded. But it's rare to use
> >        PIC/PIE on a real-time system (code size, more memory access). So a
> >        partial PIC/PIE image maybe better (see 3. TODO section). But partial
> >        PIC/PIE image may not solve this Xen start address issue.
> I like the description of the XEN_START_ADDRESS problem and solutions.
>
> For the initial implementation, a platform file is fine. We need to
> start easy.
>
> Afterwards, I think it would be far better to switch to a script that
> automatically generates XEN_START_ADDRESS from the host device tree.
> Also, if we provide a way to customize the start address via Kconfig,
> then the script that reads the device tree could simply output the right
> CONFIG_* option for Xen to build. It wouldn't even have to generate an
> header file.
Ok,I will update the proposal to create two steps for XEN_START_ADDRESS:
stage 1: Use the platform files for XEN_START_ADDRESS. If no platform
         has been selected, provide Kconfig option for users to do
         customization.
stage 2: Try to switch to use scripts that can automatically generate
         XEN_START_ADDRESS from the host device tree. But still keep
         Kconfig customization option.

And for the PIC/PIE option,I will move it to TODO list so that we
have records for future discussions.Becuase this feature for an
MPU system, its priority is not very high.

> > - ***About MPU initialization before parsing device tree***:
> >        Before Xen can start parsing information from device tree and use
> >        this information to setup MPU, Xen need an initial MPU state. This
> >        is because:
> >        1. More deterministic: Arm MPU supports background regions, if we
> >           don't configure the MPU regions and don't enable MPU. The default
> >           MPU background attributes will take effect. The default background
> >           attributes are `IMPLEMENTATION DEFINED`. That means all RAM regions
> >           may be configured to device memory and RWX. Random values in RAM or
> >           maliciously embedded data can be exploited.
> >        2. More compatible: On some Armv8-R64 platforms, if MPU is disabled,
> >           the `dc zva` instruction will make the system halt (This is one
> >           side effect of MPU background attributes, the RAM has been configured
> >           as device memory). And this instruction will be embedded in some
> >           built-in functions, like `memory set`. If we use `-ddont_use_dc` to
> >           rebuild GCC, the built-in functions will not contain `dc zva`.
> >           However, it is obviously unlikely that we will be able to recompile
> >           all GCC for ARMv8-R64.
> >
> >      - Reuse `XEN_START_ADDRESS`
> >        In the very beginning of Xen boot, Xen just need to cover a limited
> >        memory range and very few devices (actually only UART device). So we
> >        can use two MPU regions to map:
> >        1. `XEN_START_ADDRESS` to `XEN_START_ADDRESS + 2MB` or.
> >           `XEN_START_ADDRESS` to `XEN_START_ADDRESS + image_size`as
> >           normal memory.
> >        2. `UART` MMIO region base to `UART` MMIO region end to device memory.
> >        These two are enough to support Xen run in boot time. And we don't need
> >        to provide additional platform information for initial normal memory
> >        and device memory regions. In current PoC we have used this option
> >        for implementation, and it's the same as Armv8-A.
> >
> >      - Additional platform information for initial MPU state
> >        Introduce some macros to allow users to set initial normal
> >        memory regions:
> >        `ARM_MPU_NORMAL_MEMORY_START` and `ARM_MPU_NORMAL_MEMORY_END`
> >        and device memory:
> >        `ARM_MPU_DEVICE_MEMORY_START` and `ARM_MPU_DEVICE_MEMORY_END`
> >        These macros are the same platform specific information as
> >        `XEN_START_ADDRESS`, so the options#1/#2/#3 of generating
> >        `XEN_START_ADDRESS` also can be applied to these macros.
> >        ***From our current PoC work, we think these macros may***
> >        ***not be necessary. But we still place them here to see***
> >        ***whether the community will have some different scenarios***
> >        ***that we haven't considered.***
> I think it is fine for now. And their values could be automatically
> generated by the same script that will automatically generate
> XEN_START_ADDRESS from the host device tree.
Ok, we will keep current PoC "Reuse `XEN_START_ADDRESS`" for
day1 patch. We will update the proposal to address it. And place
script generatation in stage2.

> > - ***Define new system registers for compiliers***:
> >    Armv8-R64 is based on Armv8.4. That means we will use some Armv8.4
> >    specific system registers. As Armv8-R64 only have secure state, so
> >    at least, `VSTCR_EL2` and `VSCTLR_EL2` will be used for Xen. And the
> >    first GCC version that supports Armv8.4 is GCC 8.1. In addition to
> >    these, PMSA of Armv8-R64 introduced lots of MPU related system registers:
> >    `PRBAR_ELx`, `PRBARx_ELx`, `PRLAR_ELx`, `PRLARx_ELx`, `PRENR_ELx` and
> >    `MPUIR_ELx`. But the first GCC version to support these system registers
> >    is GCC 11. So we have two ways to make compilers to work properly with
> >    these system registers.
> >    1. Bump GCC version to GCC 11.
> >       The pros of this method is that, we don't need to encode these
> >       system registers in macros by ourselves. But the cons are that,
> >       we have to update Makefiles to support GCC 11 for Armv8-R64.
> >       1.1. Check the GCC version 11 for Armv8-R64.
> >       1.2. Add march=armv8r to CFLAGS for Armv8-R64.
> >       1.3. Solve the confliction of march=armv8r and mcpu=generic
> >      These changes will affect common Makefiles, not only Arm Makefiles.
> >      And GCC 11 is new, lots of toolchains and Distro haven't supported it.
> >
> >    2. Encode new system registers in macros ***(preferred)***
> >          ```
> >          /* Virtualization Secure Translation Control Register */
> >          #define VSTCR_EL2  S3_4_C2_C6_2
> >          /* Virtualization System Control Register */
> >          #define VSCTLR_EL2 S3_4_C2_C0_0
> >          /* EL1 MPU Protection Region Base Address Register encode */
> >          #define PRBAR_EL1  S3_0_C6_C8_0
> >          ...
> >          /* EL2 MPU Protection Region Base Address Register encode */
> >          #define PRBAR_EL2  S3_4_C6_C8_0
> >          ...
> >          ```
> >       If we encode all above system registers, we don't need to bump GCC
> >       version. And the common CFLAGS Xen is using still can be applied to
> >       Armv8-R64. We don't need to modify Makefiles to add specific CFLAGS.
> >       ***Notes:***
> >       ***Armv8-R AArch64 supports the A64 ISA instruction set with***
> >       ***some modifications:***
> >       ***Redefines DMB, DSB, and adds an DFB. But actually, the***
> >       ***encodings of DMB and DSB are still the same with A64.***
> >       ***And DFB is an alias of DSB #12. In this case, we think***
> >       ***we don't need a new architecture specific flag to***
> >       ***generate new instructions for Armv8-R.***
> I think that for the initial implementation either way is fine. I agree
> that macros would be better than requiring GCC 11.
Ok. We will use macros in day1. We can have a standalone patch set
to bump GCC version in the future. Based on some of the attempts we've
made, it will affect some makefile scripts of common and other
architectures.


> > ### **2.2. Changes of the initialization process**
> >
> > **A sample device tree of memory layout restriction**:
> > ```
> > chosen {
> >      ...
> >      /*
> >       * Define a section to place boot modules,
> >       * all boot modules must be placed in this section.
> >       */
> >      mpu,boot-module-section = <0x10000000 0x10000000>;
> >      /*
> >       * Define a section to cover all guest RAM. All guest RAM must be located
> >       * within this section. The pros is that, in best case, we can only have
> >       * one MPU protection region to map all guest RAM for Xen.
> >       */
> >      mpu,guest-memory-section = <0x20000000 0x30000000>;
> >      /*
> >       * Define a memory section that can cover all device memory that
> >       * will be used in Xen.
> >       */
> >      mpu,device-memory-section = <0x80000000 0x7ffff000>;
> >      /* Define a section for Xen heap */
> >      xen,static-mem = <0x50000000 0x20000000>;
> >
> >      domU1 {
> >          ...
> >          #xen,static-mem-address-cells = <0x01>;
> >          #xen,static-mem-size-cells = <0x01>;
> >          /* Statically allocated guest memory, within mpu,guest-memory-section 
> > */
> >          xen,static-mem = <0x30000000 0x1f000000>;
> >
> >          module@11000000 {
> >              compatible = "multiboot,kernel\0multiboot,module";
> >              /* Boot module address, within mpu,boot-module-section */
> >              reg = <0x11000000 0x3000000>;
> >              ...
> >          };
> >
> >          module@10FF0000 {
> >                  compatible = "multiboot,device-tree\0multiboot,module";
> >                  /* Boot module address, within mpu,boot-module-section */
> >                  reg = <0x10ff0000 0x10000>;
> >                  ...
> >          };
> >      };
> > };
> > ```
> > It's little hard for users to compose such a device tree by hand. Based
> > on the discussion of Draft-A, Xen community suggested users to use some
> > tools like 
> > [imagebuilder](https://gitlab.com/ViryaOS/imagebuilder/-/blob/master/scripts/uboot-script-gen#L390)
> >  to generate the above device tree properties.
> > Please goto TODO#3.3 section to get more details of this suggestion.
> Yes, I think we'll need an ImageBuilder script to populate these entries
> automatically. With George's help, I moved ImageBuilder to Xen Project.
> This is the new repository: https://gitlab.com/xen-project/imagebuilder
>
> The script to generate mpu,boot-module-section and the other mpu
> addresses could be the same ImageBuilder script that generates also
> XEN_START_ADDRESS.
That's great, I will update the link in proposal.

> > ### **2.4. Changes of memory management**
> > Xen is coupled with VMSA, in order to port Xen to Armv8-R64, we have to
> > decouple Xen from VMSA. And give Xen the ablity to manage memory in PMSA.
...          ```
> > ### **2.5. Changes of guest management**
...
> > ### **2.6. Changes of exception trap**
...
> > ### **2.5. Changes of device driver**
...
> > ## 3. TODO
...
> > ### 3.1. Alternative framework support
...
> > ### 3.2. Xen Event Channel Support
...
> > ### 3.3. Xen Partial PIC/PIE
...
> > ### 3.4. A tool to generate Armv8-R Xen device tree
> > 1. Use a tool to generate above device tree property.
> >     This tool will have some similar inputs as below:
> >     ---
> >     DEVICE_TREE="fvp_baremetal.dtb"
> >     XEN="4.16-2022.1/xen"
> >
> >     NUM_DOMUS=1
> >     DOMU_KERNEL[0]="4.16-2022.1/Image-domU"
> >     DOMU_RAMDISK[0]="4.16-2022.1/initrd.cpio"
> >     DOMU_PASSTHROUGH_DTB[0]="4.16-2022.1/passthrough-example-dev.dtb"
> >     DOMU_RAM_BASE[0]=0x30000000
> >     DOMU_RAM_SIZE[0]=0x1f000000
> >     ---
> >     Using above inputs, the tool can generate a device tree similar as
> >     we have described in sample.
> >
> >     - `mpu,guest-memory-section`:
> >     This section will cover all guests' RAM (`xen,static-mem` defined regions
> >     in all DomU nodes). All guest RAM must be located within this section.
> >     In the best case, we can only have one MPU protection region to map all
> >     guests' RAM for Xen.
> >
> >     If users set `DOMU_RAM_BASE` and `DOMU_RAM_SIZE`, these will be converted
> >     to the base and size of `xen,static-mem`. This tool will scan all
> >     `xen, static-mem` in DomU nodes to determin the base and size of
> >     `mpu,guest-memory-section`. If there is any other kind of memory usage
> >     has been detected in this section, this tool can report an error.
> >     Except build time check, Xen also need to do runtime check to prevent a
> >     bad device tree that generated by malicious tools.
> >
> >     If users set `DOMU_RAM_SIZE` only, this will be converted to the size of
> >     `xen,static-mem` only. Xen will allocate the guest memory in runtime, but
> >     not from Xen heap. `mpu,guest-memory-section` will be caculated in runtime
> >     too. The property in device tree doesn't need or will be ignored by Xen.
> I am fine with this. You should also know that there was a recent
> discussion about adding something like:
>
> # address size address size ...
> DOMU_STATIC_MEM_RANGES[0]="0xe000000 0x1000000 0xa0000000 0x30000000"
>
> to the ImageBuilder config file.
Thanks for this update : )

> >     - `mpu,boot-module-section`:
> >     This section will be used to store the boot modules like DOMU_KERNEL,
> >     DOMU_RAMDISK, and DOMU_PASSTHROUGH_DTB. Xen keeps all boot modules in
> >     this section to meet the requirement of DomU restart on Armv8-R. In
> >     current stage, we don't have a privilege domain like Dom0 that can
> >     access filesystem to reload DomU images.
> >
> >     And in current Xen code, the base and size are mandatory for boot modules
> >     If users don't specify the base of each boot module, the tool will
> >     allocte a base for each module. And the tool will generate the
> >     `mpu,boot-module-section` region, when it finishs boot module memory
> >     allocation.
> >
> >     Users also can specify the base and size of each boot module, these will
> >     be converted to the base and size of module's `reg` directly. The tool
> >     will scan all modules `reg` in DomU nodes to generate the base and size of
> >     `mpu,boot-module-section`. If there is any kind of other memory usage
> >     has been detected in this section, this tool can report an error.
> >     Except build time check, Xen also need to do runtime check to prevent a
> >     bad device tree that generated by malicious tools.
> Xen should always check for the validity of its input. However I should
> point out that there is no "malicious tool" in this picture because a
> malicious entity with access to the tool would also have access to Xen
> directly, so they might as well replace the Xen binary.
Ok, I will drop the "malicious tools". But I think the "bug tools" still
can be possible : )

> >     - `mpu,device-memory-section`:
> >     This section will cover all device memory that will be used in Xen. Like
> >     `UART`, `GIC`, `SMMU` and other devices. We haven't considered multiple
> >     `mpu,device-memory-section` scenarios. The devices' memory and RAM are
> >     interleaving in physical address space, it would be required to use
> >     multiple `mpu,device-memory-section` to cover all devices. This layout
> >     is common on Armv8-A system, especially in server. But it's rare in
> >     Armv8-R. So in current stage, we don't want to allow multiple
> >     `mpu,device-memory-section`. The tool can scan baremetal device tree
> >     to sort all devices' memory ranges. And calculate a proper region for
> >     `mpu,device-memory-section`. If it find Xen need multiple
> >     `mpu,device-memory-section`, it can report an unsupported error.
> >
> > 2. Use a tool to generate device tree property and platform files
> >     This opinion still uses the same inputs as opinion#1. But this tool only
> >     generates `xen,static-mem` and `module` nodes in DomU nodes, it will not
> >     generate `mpu,guest-memory-section`, `mpu,boot-module-section` and
> >     `mpu,device-memory-section` properties in device tree. This will
> >     generate following macros:
> >     `MPU_GUEST_MEMORY_SECTION_BASE`, `MPU_GUEST_MEMORY_SECTION_SIZE`
> >     `MPU_BOOT_MODULE_SECTION_BASE`, `MPU_BOOT_MODULE_SECTION_SIZE`
> >     `MPU_DEVICE_MEMORY_SECTION_BASE`, `MPU_DEVICE_MEMORY_SECTION_SIZE`
> >     in platform files in build time. In runtime, Xen will skip the device
> >     tree parsing for `mpu,guest-memory-section`, `mpu,boot-module-section`
> >     and `mpu,device-memory-section`. And instead Xen will use these macros
> >     to do runtime check.
> >     But, this also means these macros only exist in local build system,
> >     these macros will not be maintained in Xen repo.
> Yes this makes sense to me.
>
> I think we should add both scripts to the imagebuilder repository. This
> way, they could share code easily, and we can keep the documentation in
> a single place.
Can I understand your comments like, we can support above two options.
Users can select to use either way. If the select the option#2 script,
MPU_GUEST_MEMORY_SECTION_BASE will be detected by Xen code, and Xen
can bypass the device tree parser. If Xen can't detected 
MPU_GUEST_MEMORY_SECTION_BASE, Xen can treat that users selected to
use option#1 scripts, Xen will do DT parser.

Cheers,
Wei Chen