Re: DMA restriction and NUMA node number

[Julien Grall <julien@xxxxxxx>]

(+Andrew, Jan, Roger)

On 13/07/2021 04:19, Wei Chen wrote:
> Hi,
>
> I am doing some NUMA testing on Xen. And I find the DMA restriction is
> based on NUMA node number [1].
>      if ( !dma_bitsize && (num_online_nodes() > 1) )
>          dma_bitsize = arch_get_dma_bitsize();
>
> On Arm64, we will set dma_bitsize [2] to 0, that means we don't need to
> reserve DMA memory. But when num_online_nodes > 1, the dma_bitsize
> will override to 32. This may be caused by the Arm64 version
> arch_get_dma_bitsize, it may be a simple implementation and not NUMA
> aware.
>
> But I still quite curious about why DMA restriction depends on NUMA
> node number. In Arm64, dma_bitsize does not change when the NUMA node
> changes. So we didn't expect arch_get_dma_bitsize to be called here.
>
> I copied Keir's commit message from 2008. It seems this code was considered
> only for x86, when he was working on it. But I'm not an x86 expert, so I
> hope Xen x86 folks can give some help. Understanding this will help us to
It is best to CCed the relevant person so they know you have requested 
there input. I have added the x86 maintainers in the thread.
> do some adaptations to Arm in subsequent modifications : )
>
> commit accacb43cb7f16e9d1d8c0e58ea72c9d0c32cec2
> Author: Keir Fraser <keir.fraser@xxxxxxxxxx>
> Date:   Mon Jul 28 16:40:30 2008 +0100
>
>      Simplify 'dma heap' logic.
>
>      1. Only useful for NUMA systems, so turn it off on non-NUMA systems by
>         default.
>      2. On NUMA systems, by default relate the DMA heap size to NUMA node 0
>         memory size (so that not all of node 0's memory ends up being 'DMA
>         heap').
>      3. Remove the 'dma emergency pool'. It's less useful now that running
>         out of low memory isn;t as fatal as it used to be (e.g., when we
>         needed to be able to allocate low-memory PAE page directories).
>
> [1] 
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/common/page_alloc.c;h=958ba0cd9256c8270e38585d272be2bf5cc0679e;hb=refs/heads/master#l1876
> [2] 
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/common/page_alloc.c;h=958ba0cd9256c8270e38585d272be2bf5cc0679e;hb=refs/heads/master#l226
Cheers,

--
Julien Grall