RE: [PATCH v2 00/10] Device tree based NUMA support for Arm - Part#1

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi,

I am sorry, I had added a wrong address--to=xen-devel@xxxxxxxxxxxxxxxxxxxx
in this series' recipient address by mistake. When you reply to this series,
please remember to remove it. I'm sorry for the inconvenience again!

Cheers,
Wei Chen

> -----Original Message-----
> From: Wei Chen <wei.chen@xxxxxxx>
> Sent: 2022年4月18日 17:07
> To: --to=xen-devel@xxxxxxxxxxxxxxxxxxxx; xen-devel@xxxxxxxxxxxxxxxxxxxx
> Cc: nd <nd@xxxxxxx>; Wei Chen <Wei.Chen@xxxxxxx>; Stefano Stabellini
> <sstabellini@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Jan Beulich
> <jbeulich@xxxxxxxx>
> Subject: [PATCH v2 00/10] Device tree based NUMA support for Arm - Part#1
> 
> (The Arm device tree based NUMA support patch set contains 35
> patches. In order to make stuff easier for reviewers, I split
> them into 3 parts:
> 1. Preparation. I have re-sorted the patch series. And moved
>    independent patches to the head of the series.
> 2. Move generically usable code from x86 to common.
> 3. Add new code to support Arm.
> 
> This series only contains the first part patches.
> The full patch list can be found in this cover letter.)
> 
> Xen memory allocation and scheduler modules are NUMA aware.
> But actually, on x86 has implemented the architecture APIs
> to support NUMA. Arm was providing a set of fake architecture
> APIs to make it compatible with NUMA awared memory allocation
> and scheduler.
> 
> Arm system was working well as a single node NUMA system with
> these fake APIs, because we didn't have multiple nodes NUMA
> system on Arm. But in recent years, more and more Arm devices
> support multiple nodes NUMA system.
> 
> So now we have a new problem. When Xen is running on these Arm
> devices, Xen still treat them as single node SMP systems. The
> NUMA affinity capability of Xen memory allocation and scheduler
> becomes meaningless. Because they rely on input data that does
> not reflect real NUMA layout.
> 
> Xen still think the access time for all of the memory is the
> same for all CPUs. However, Xen may allocate memory to a VM
> from different NUMA nodes with different access speeds. This
> difference can be amplified in workloads inside VM, causing
> performance instability and timeouts.
> 
> So in this patch series, we implement a set of NUMA API to use
> device tree to describe the NUMA layout. We reuse most of the
> code of x86 NUMA to create and maintain the mapping between
> memory and CPU, create the matrix between any two NUMA nodes.
> Except ACPI and some x86 specified code, we have moved other
> code to common. In next stage, when we implement ACPI based
> NUMA for Arm64, we may move the ACPI NUMA code to common too,
> but in current stage, we keep it as x86 only.
> 
> This patch serires has been tested and booted well on one
> Arm64 NUMA machine and one HPE x86 NUMA machine.
> 
> ---
> Part1 v1->v2:
> 1. Move independent patches from later to early of this series.
> 2. Drop the copy of EFI stub.c from Arm. Share common codes of
>    x86 EFI stub for Arm.
> 3. Use CONFIG_ARM_EFI to replace CONFIG_EFI and remove help text
>    and make CONFIG_ARM_EFI invisible.
> 4. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
> 5. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
> 6. Extend the description of Arm's workaround for reserve DMA
>    allocations to avoid the same discussion every time for
>    arch_have_default_dmazone.
> 7. Update commit messages.
> 
> Wei Chen (10):
>   xen/arm: Print a 64-bit number in hex from early uart
>   xen/x86: move reusable EFI stub functions from x86 to common
>   xen/arm: add CONFIG_ARM_EFI to stub EFI API
>   xen/arm: Keep memory nodes in device tree when Xen boots from EFI
>   xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid
>   xen: introduce an arch helper for default dma zone status
>   xen: decouple NUMA from ACPI in Kconfig
>   xen/arm: use !CONFIG_NUMA to keep fake NUMA API
>   xen/x86: use paddr_t for addresses in NUMA node structure
>   xen/x86: add detection of memory interleaves for different nodes
> 
> PART2:
>   xen/x86: introduce a helper to update memory hotplug end
>   xen/x86: Use enumerations to indicate NUMA status
>   xen/x86: move generically usable NUMA code from x86 to common
>   xen/x86: use arch_get_memory_map to get information from E820 map
>   xen/x86: move NUMA scan nodes codes from x86 to common
>   xen/x86: rename bad_srat to numa_bad
> 
> PART3:
>   xen: rename acpi_scan_nodes to numa_scan_nodes
>   xen: introduce a Kconfig option to configure NUMA nodes number
>   xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
>   xen/arm: implement helpers to get and update NUMA status
>   xen/arm: implement node distance helpers for Arm
>   xen/arm: use arch_get_memory_map to memory bank from bootinfo
>   xen/arm: build NUMA cpu_to_node map in dt_smp_init_cpus
>   xen/arm: Add boot and secondary CPU to NUMA system
>   xen/arm: stub mem_hotplug_update_boundary for Arm
>   xen/arm: introduce a helper to parse device tree processor node
>   xen/arm: introduce a helper to parse device tree memory node
>   xen/arm: introduce a helper to parse device tree NUMA distance map
>   xen/arm: unified entry to parse all NUMA data from device tree
>   xen/arm: keep guest still be NUMA unware
>   xen/arm: enable device tree based NUMA in system init
>   xen/arm: implement a dummy 1:1 node to pxm mapping
>   xen/arm: use CONFIG_NUMA to gate node_online_map in smpboot
>   xen/arm: Provide Kconfig options for Arm to enable NUMA
>   docs: update numa command line to support Arm
> 
>  xen/arch/arm/Kconfig                    |  5 ++
>  xen/arch/arm/Makefile                   |  2 +-
>  xen/arch/arm/arm64/head.S               | 12 ++--
>  xen/arch/arm/bootfdt.c                  |  8 ++-
>  xen/arch/arm/efi/Makefile               |  5 ++
>  xen/arch/arm/efi/efi-boot.h             | 25 ---------
>  xen/arch/arm/include/asm/numa.h         |  6 ++
>  xen/arch/x86/Kconfig                    |  2 +-
>  xen/arch/x86/efi/Makefile               |  4 +-
>  xen/arch/x86/efi/{stub.c => stub-x86.c} | 37 -------------
>  xen/arch/x86/include/asm/config.h       |  1 -
>  xen/arch/x86/include/asm/numa.h         | 15 ++---
>  xen/arch/x86/numa.c                     | 30 +++++-----
>  xen/arch/x86/srat.c                     | 74 +++++++++++++++++++------
>  xen/common/Kconfig                      |  3 +
>  xen/common/efi/stub.c                   | 38 +++++++++++++
>  xen/common/page_alloc.c                 |  2 +-
>  xen/drivers/acpi/Kconfig                |  3 +-
>  xen/drivers/acpi/Makefile               |  2 +-
>  xen/include/xen/numa.h                  |  2 +
>  20 files changed, 162 insertions(+), 114 deletions(-)
>  rename xen/arch/x86/efi/{stub.c => stub-x86.c} (71%)
>  create mode 100644 xen/common/efi/stub.c
> 
> --
> 2.25.1