[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Proposal for Porting Xen to Armv8-R64 - DraftA



Hi Wei,

Thank you for sending the proposal. Please find some comments below.

On 24/02/2022 06:01, Wei Chen wrote:
# Proposal for Porting Xen to Armv8-R64

This proposal will introduce the PoC work of porting Xen to Armv8-R64,
which includes:
- The changes of current Xen capability, like Xen build system, memory
   management, domain management, vCPU context switch.
- The expanded Xen capability, like static-allocation and direct-map.

***Notes:***
1. ***This proposal only covers the work of porting Xen to Armv8-R64***
    ***single CPU. Xen SMP support on Armv8-R64 relates to Armv8-R***
    ***Trusted-Frimware (TF-R). This is an external dependency,***
    ***so we think the discussion of Xen SMP support on Armv8-R64***
    ***should be started when single-CPU support is complete.***

I agree that we should first focus on single-CPU support.

2. ***This proposal will not touch xen-tools. In current stage,***
    ***Xen on Armv8-R64 only support dom0less, all guests should***
    ***be booted from device tree.***

Make sense. I actually expect some issues in the way xen-tools would need to access memory of the domain that is been created.

[...]

### 1.2. Xen Challenges with PMSA Virtualization
Xen is PMSA unaware Type-1 Hypervisor, it will need modifications to run
with an MPU and host multiple guest OSes.

- No MMU at EL2:
     - No EL2 Stage 1 address translation
         - Xen provides fixed ARM64 virtual memory layout as basis of EL2
           stage 1 address translation, which is not applicable on MPU system,
           where there is no virtual addressing. As a result, any operation
           involving transition from PA to VA, like ioremap, needs modification
           on MPU system.
     - Xen's run-time addresses are the same as the link time addresses.
         - Enable PIC (position-independent code) on a real-time target
           processor probably very rare.

Aside the assembly boot code and UEFI stub, Xen already runs at the same address as it was linked.

     - Xen will need to use the EL2 MPU memory region descriptors to manage
       access permissions and attributes for accesses made by VMs at EL1/0.
         - Xen currently relies on MMU EL1 stage 2 table to manage these
           accesses.
- No MMU Stage 2 translation at EL1:
     - A guest doesn't have an independent guest physical address space
     - A guest can not reuse the current Intermediate Physical Address
       memory layout
     - A guest uses physical addresses to access memory and devices
     - The MPU at EL2 manages EL1 stage 2 access permissions and attributes
- There are a limited number of MPU protection regions at both EL2 and EL1:
     - Architecturally, the maximum number of protection regions is 256,
       typical implementations have 32.
     - By contrast, Xen does not need to consider the number of page table
       entries in theory when using MMU.
- The MPU protection regions at EL2 need to be shared between the hypervisor
   and the guest stage 2.
     - Requires careful consideration - may impact feature 'fullness' of both
       the hypervisor and the guest
     - By contrast, when using MMU, Xen has standalone P2M table for guest
       stage 2 accesses.

[...]

- ***Define new system registers for compilers***:
   Armv8-R64 is based on Armv8.4. That means we will use some Armv8.4
   specific system registers. As Armv8-R64 only have secure state, so
   at least, `VSTCR_EL2` and `VSCTLR_EL2` will be used for Xen. And the
   first GCC version that supports Armv8.4 is GCC 8.1. In addition to
   these, PMSA of Armv8-R64 introduced lots of MPU related system registers:
   `PRBAR_ELx`, `PRBARx_ELx`, `PRLAR_ELx`, `PRLARx_ELx`, `PRENR_ELx` and
   `MPUIR_ELx`. But the first GCC version to support these system registers
   is GCC 11. So we have two ways to make compilers to work properly with
   these system registers.
   1. Bump GCC version to GCC 11.
      The pros of this method is that, we don't need to encode these
      system registers in macros by ourselves. But the cons are that,
      we have to update Makefiles to support GCC 11 for Armv8-R64.
      1.1. Check the GCC version 11 for Armv8-R64.
      1.2. Add march=armv8r to CFLAGS for Armv8-R64.
      1.3. Solve the confliction of march=armv8r and mcpu=generic
     These changes will affect common Makefiles, not only Arm Makefiles.
     And GCC 11 is new, lots of toolchains and Distro haven't supported it.

I agree that forcing to use GCC11 is not a good idea. But I am not sure to understand the problem with the -march=.... Ultimately, shouldn't we aim to build Xen ARMv8-R with -march=armv8r?

[...]

### **2.2. Changes of the initialization process**
In general, we still expect Armv8-R64 and Armv8-A64 to have a consistent
initialization process. In addition to some architecture differences, there
is no more than reusable code that we will distinguish through CONFIG_ARM_MPU
or CONFIG_ARM64_V8R. We want most of the initialization code to be reusable
between Armv8-R64 and Armv8-A64.

- We will reuse the original head.s and setup.c of Arm. But replace the
   MMU and page table operations in these files with configuration operations
   for MPU and MPU regions.

- We provide a boot-time MPU configuration. This MPU configuration will
   support Xen to finish its initialization. And this boot-time MPU
   configuration will record the memory regions that will be parsed from
   device tree.

   In the end of Xen initialization, we will use a runtime MPU configuration
   to replace boot-time MPU configuration. The runtime MPU configuration will
   merge and reorder memory regions to save more MPU regions for guests.
   
![img](https://drive.google.com/uc?export=view&id=1wTFyK2XfU3lTlH1PqRDoacQVTwUtWIGU)

- Defer system unpausing domain.
   When Xen initialization is about to end, Xen unpause guests created
   during initialization. But this will cause some issues. The unpause
   action occurs before free_init_memory, however the runtime MPU configuration
   is built after free_init_memory.

I was half expecting that free_init_memory() would not be called for Xen Armv8R.


   So if the unpaused guests start executing the context switch at this
   point, then its MPU context will base on the boot-time MPU configuration.

Can you explain why you want to switch the MPU configuration that late?

   Probably it will be inconsistent with runtime MPU configuration, this
   will cause unexpected problems (This may not happen in a single core
   system, but on SMP systems, this problem is foreseeable, so we hope to
   solve it at the beginning).

[...]

### **2.4. Changes of memory management**
Xen is coupled with VMSA, in order to port Xen to Armv8-R64, we have to
decouple Xen from VMSA. And give Xen the ability to manage memory in PMSA.

1. ***Use buddy allocator to manage physical pages for PMSA***
    From the view of physical page, PMSA and VMSA don't have any difference.
    So we can reuse buddy allocator on Armv8-R64 to manage physical pages.
    The difference is that, in VMSA, Xen will map allocated pages to virtual
    addresses. But in PMSA, Xen just convert the pages to physical address.

2. ***Can not use virtual address for memory management***
    As Armv8-R64 only has PMSA in EL2, Xen loses the ability of using virtual
    address to manage memory. This brings some problems, some virtual address
    based features could not work well on Armv8-R64, like `FIXMAP`, 
`vmap/vumap`,
    `ioremap` and `alternative`.

    But the functions or macros of these features are used in lots of common
    code. So it's not good to use `#ifdef CONFIG_ARM_MPU` to gate relate code
    everywhere. In this case, we propose to use stub helpers to make the changes
    transparently to common code.
    1. For `FIXMAP`, we will use `0` in `FIXMAP_ADDR` for all fixmap operations.
       This will return physical address directly of fixmapped item.
    2. For `vmap/vumap`, we will use some empty inline stub helpers:
         ```
         static inline void vm_init_type(...) {}
         static inline void *__vmap(...)
         {
             return NULL;
         }
         static inline void vunmap(const void *va) {}
         static inline void *vmalloc(size_t size)
         {
             return NULL;
         }
         static inline void *vmalloc_xen(size_t size)
         {
             return NULL;
         }
         static inline void vfree(void *va) {}
         ```

    3. For `ioremap`, it depends on `vmap`. As we have make `vmap` to always
       return `NULL`, they could not work well on Armv8-R64 without changes.
       `ioremap` will return input address directly.
         ```
         static inline void *ioremap_attr(...)
         {
             /* We don't have the ability to change input PA cache attributes */
OOI, who will set them?

             if ( CACHE_ATTR_need_change )
                 return NULL;
             return (void *)pa;
         }
         static inline void __iomem *ioremap_nocache(...)
         {
             return ioremap_attr(start, len, PAGE_HYPERVISOR_NOCACHE);
         }
         static inline void __iomem *ioremap_cache(...)
         {
             return ioremap_attr(start, len, PAGE_HYPERVISOR);
         }
         static inline void __iomem *ioremap_wc(...)
         {
             return ioremap_attr(start, len, PAGE_HYPERVISOR_WC);
         }
         void *ioremap(...)
         {
             return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
         }

         ```
     4. For `alternative`, it depends on `vmap` too.

The only reason we depend on vmap() is because the map the sections *text read-only and we enforce WnX. For VMSA, it would be possible to avoid vmap() with some rework. I don't know for PMSA.

We will simply disable
        it on Armv8-R64 in current stage. How to implement `alternative`
        on Armv8-R64 is better to be discussed after basic functions of Xen
        on Armv8-R64 work well.
alternative are mostly helpful to handle errata or enable features that are not present on all CPUs. I wouldn't expect this to be necessary at the beginning. In fact, on Arm, it was introduced > 4 years after the initial port :).

[...]

### **2.5. Changes of device driver**
1. Because Armv8-R64 only has single secure state, this will affect some
devices that have two secure state, like GIC. But fortunately, most
vendors will not link a two secure state GIC to Armv8-R64 processors.
Current GIC driver can work well with single secure state GIC for Armv8-R64.
2. Xen should use secure hypervisor timer in Secure EL2. We will introduce
a CONFIG_ARM_SECURE_STATE to make Xen to use secure registers for timer.

### **2.7. Changes of virtual device**
Currently, we only support pass-through devices in guest. Because event
channel, xen-bus, xen-storage and other advanced Xen features haven't been
enabled in Armv8-R64.

That's fine. I expect to require quite a bit of work to move from Xen sharing the pages (e.g. like for grant-tables) to the guest sharing pages.

Cheers,

--
Julien Grall



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.