Re: [Xen-devel] arm64: Approach for DT based NUMA and issues

[Vijay Kilari <vijay.kilari@xxxxxxxxx>]

On Thu, Mar 2, 2017 at 6:22 PM, Julien Grall <julien.grall@xxxxxxx> wrote:
> Hello Vijay,
>
>
> On 02/03/17 12:39, Vijay Kilari wrote:
>>
>> On Fri, Dec 16, 2016 at 3:48 PM, Dario Faggioli
>> <dario.faggioli@xxxxxxxxxx> wrote:
>>>
>>> On Fri, 2016-12-16 at 09:40 +0000, Julien Grall wrote:
>>>>
>>>> Hi Vijay,
>>>> On 16/12/2016 07:39, Vijay Kilari wrote:
>>>>>
>>>>> If we drop numa-node-id from memory node generated to dom0, then
>>>>> dom0 will
>>>>> assume all the memory is from node0. So eventually node1 device
>>>>> intialization fails.
>>>>
>>>>
>>>> I suggested to drop the property numa-node-id from every node (not
>>>> only
>>>> memory one). So DOM0 will think it is running a non-NUMA platform.
>>>>
>>>>  From my knowledge this is working on x86, and I don't understand
>>>> why
>>>> this would be an issue on ARM. If you think the device may not work,
>>>> please explain why.
>>>>
>>> Yes, I confirm that what you said works and any x86 NUMA system I've
>>> seen.
>>>
>>> AFAIUI, since Vijay is talking about "devices", the x86 equivalent of
>>> what would be an "IONUMA system", i.e. a platform where I/O devices are
>>> physically attached to more than just one I/O hub, which in turn are
>>> attached to different nodes.
>>> I don't have first hand experience with these systems on the x86 world,
>>> but I'm quite sure they also function with the configuration Julien is
>>> suggesting to use.
>>>
>>> Boris did some work to _improve_ the situation (namely, to make it
>>> possible for *Xen* to report to the toolstack, to which NUMA node a
>>> specific device is attached to). But:
>>>  - things were working already before this
>>>  - that does involve Xen and toolstack, while Dom0 remains totally
>>>    NUMA _unaware_.
>>>
>>> And I indeed think that doing what Julien says (i.e., keep dom0 NUMA-
>>> ignorant) is, if possible, best as a first step.
>>
>>
>>
>> Sorry for late reply. I want to confirm only after checking if this
>> approach works.
>>
>> Yes for first step implementation this fine.
>> For now, our platform does not have any such restrictions that memory
>> for IO device
>> should be always local.
>> Also, the issue is seen with SMMU, which is going to be hidden from Dom0.
>> I have mentioned this restriction in my NUMA RFC patch commit.
>
>
> Can you detail the restrictions with SMMU? Which memory should be allocated
> to the correct node? Stage-2 page tables?

The issue is seen when DOm0 is booted with SMMU, on NUMA platform with DT.
SMMU driver in Linux uses devm_* calls to allocate memory for its structures.
The SMMU on node1 tries to allocate memory from node1 memory.
So if there is no node1 memory in Dom0 then driver panics.(Xen does not expose
NUMA memory info to DOM0).

The solution proposed by you is to drop numa-node-id from all DT nodes
and hence node1 SMMU will allocate memory from node 0 and boots fine.

So, I said since SMMU will be hidden from DOM0, the issue does not occur at all.

Regards
Vijay

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel