RE: [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Stefano,

> -----Original Message-----
> From: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> Sent: 2021年8月26日 8:09
> To: Wei Chen <Wei.Chen@xxxxxxx>
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx; sstabellini@xxxxxxxxxx; julien@xxxxxxx;
> jbeulich@xxxxxxxx; Bertrand Marquis <Bertrand.Marquis@xxxxxxx>;
> andrew.cooper3@xxxxxxxxxx
> Subject: Re: [XEN RFC PATCH 00/40] Add device tree based NUMA support to
> Arm64
> 
> Thanks for the big contribution!
> 
> I just wanted to let you know that the series passed all the gitlab-ci
> build tests without issues.
> 
> The runtime tests originally failed due to unrelated problems (there was
> a Debian testing upgrade that broke Gitlab-CI.) I fix the underlying
> issue and restarted the failed tests and now they passed.
> 
> This is the pipeline:
> https://gitlab.com/xen-project/patchew/xen/-/pipelines/351484940
> 
> There are still two runtime x86 tests that fail but I don't think the
> failures are related to your series.
> 
> 

Thanks for testing this series : )

> On Wed, 11 Aug 2021, Wei Chen wrote:
> > Xen memory allocation and scheduler modules are NUMA aware.
> > But actually, on x86 has implemented the architecture APIs
> > to support NUMA. Arm was providing a set of fake architecture
> > APIs to make it compatible with NUMA awared memory allocation
> > and scheduler.
> >
> > Arm system was working well as a single node NUMA system with
> > these fake APIs, because we didn't have multiple nodes NUMA
> > system on Arm. But in recent years, more and more Arm devices
> > support multiple nodes NUMA system. Like TX2, some Hisilicon
> > chips and the Ampere Altra.
> >
> > So now we have a new problem. When Xen is running on these Arm
> > devices, Xen still treat them as single node SMP systems. The
> > NUMA affinity capability of Xen memory allocation and scheduler
> > becomes meaningless. Because they rely on input data that does
> > not reflect real NUMA layout.
> >
> > Xen still think the access time for all of the memory is the
> > same for all CPUs. However, Xen may allocate memory to a VM
> > from different NUMA nodes with different access speeds. This
> > difference can be amplified in workloads inside VM, causing
> > performance instability and timeouts.
> >
> > So in this patch series, we implement a set of NUMA API to use
> > device tree to describe the NUMA layout. We reuse most of the
> > code of x86 NUMA to create and maintain the mapping between
> > memory and CPU, create the matrix between any two NUMA nodes.
> > Except ACPI and some x86 specified code, we have moved other
> > code to common. In next stage, when we implement ACPI based
> > NUMA for Arm64, we may move the ACPI NUMA code to common too,
> > but in current stage, we keep it as x86 only.
> >
> > This patch serires has been tested and booted well on one
> > Arm64 NUMA machine and one HPE x86 NUMA machine.
> >
> > Hongda Deng (2):
> >   xen/arm: return default DMA bit width when platform is not set
> >   xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0
> >
> > Wei Chen (38):
> >   tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
> >   xen/arm: Print a 64-bit number in hex from early uart
> >   xen/x86: Initialize memnodemapsize while faking NUMA node
> >   xen: decouple NUMA from ACPI in Kconfig
> >   xen/arm: use !CONFIG_NUMA to keep fake NUMA API
> >   xen/x86: Move NUMA memory node map functions to common
> >   xen/x86: Move numa_add_cpu_node to common
> >   xen/x86: Move NR_NODE_MEMBLKS macro to common
> >   xen/x86: Move NUMA nodes and memory block ranges to common
> >   xen/x86: Move numa_initmem_init to common
> >   xen/arm: introduce numa_set_node for Arm
> >   xen/arm: set NUMA nodes max number to 64 by default
> >   xen/x86: move NUMA API from x86 header to common header
> >   xen/arm: Create a fake NUMA node to use common code
> >   xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
> >   xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
> >   xen: fdt: Introduce a helper to check fdt node type
> >   xen/arm: implement node distance helpers for Arm64
> >   xen/arm: introduce device_tree_numa as a switch for device tree NUMA
> >   xen/arm: introduce a helper to parse device tree processor node
> >   xen/arm: introduce a helper to parse device tree memory node
> >   xen/arm: introduce a helper to parse device tree NUMA distance map
> >   xen/arm: unified entry to parse all NUMA data from device tree
> >   xen/arm: Add boot and secondary CPU to NUMA system
> >   xen/arm: build CPU NUMA node map while creating cpu_logical_map
> >   xen/x86: decouple nodes_cover_memory with E820 map
> >   xen/arm: implement Arm arch helpers Arm to get memory map info
> >   xen: move NUMA memory and CPU parsed nodemasks to common
> >   xen/x86: move nodes_cover_memory to common
> >   xen/x86: make acpi_scan_nodes to be neutral
> >   xen: export bad_srat and srat_disabled to extern
> >   xen: move numa_scan_nodes from x86 to common
> >   xen: enable numa_scan_nodes for device tree based NUMA
> >   xen/arm: keep guest still be NUMA unware
> >   xen: introduce an arch helper to do NUMA init failed fallback
> >   xen/arm: enable device tree based NUMA in system init
> >   xen/x86: move numa_setup to common to support NUMA switch in command
> >     line
> >   xen/x86: move dump_numa info hotkey to common
> >
> >  tools/libs/util/libxlu_pci.c    |   3 +-
> >  xen/arch/arm/Kconfig            |  10 +
> >  xen/arch/arm/Makefile           |   2 +
> >  xen/arch/arm/arm64/head.S       |   9 +-
> >  xen/arch/arm/bootfdt.c          |   8 +-
> >  xen/arch/arm/domain_build.c     |  17 +-
> >  xen/arch/arm/efi/efi-boot.h     |  25 --
> >  xen/arch/arm/numa.c             | 162 +++++++++
> >  xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
> >  xen/arch/arm/platform.c         |   4 +-
> >  xen/arch/arm/setup.c            |  14 +
> >  xen/arch/arm/smpboot.c          |  37 +-
> >  xen/arch/x86/Kconfig            |   2 +-
> >  xen/arch/x86/numa.c             | 421 +----------------------
> >  xen/arch/x86/srat.c             | 147 +-------
> >  xen/common/Kconfig              |   3 +
> >  xen/common/Makefile             |   1 +
> >  xen/common/libfdt/fdt_ro.c      |  15 +
> >  xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
> >  xen/common/page_alloc.c         |   2 +-
> >  xen/drivers/acpi/Kconfig        |   3 +-
> >  xen/drivers/acpi/Makefile       |   2 +-
> >  xen/include/asm-arm/numa.h      |  33 ++
> >  xen/include/asm-arm/setup.h     |   6 +
> >  xen/include/asm-x86/acpi.h      |   4 -
> >  xen/include/asm-x86/config.h    |   1 -
> >  xen/include/asm-x86/numa.h      |  65 +---
> >  xen/include/asm-x86/setup.h     |   1 -
> >  xen/include/xen/libfdt/libfdt.h |  25 ++
> >  xen/include/xen/nodemask.h      |   2 +
> >  xen/include/xen/numa.h          |  80 +++++
> >  31 files changed, 1325 insertions(+), 659 deletions(-)
> >  create mode 100644 xen/arch/arm/numa.c
> >  create mode 100644 xen/arch/arm/numa_device_tree.c
> >  create mode 100644 xen/common/numa.c
> >
> > --
> > 2.25.1
> >