Re: xen-swiotlb issue when NVMe driver is enabled in Dom0 on ARM

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hi Stefano,

> On 18 Apr 2022, at 9:04 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
>
> On Sun, 17 Apr 2022, Rahul Singh wrote:
>>> On 15 Apr 2022, at 6:40 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
>>> On Fri, 15 Apr 2022, Christoph Hellwig wrote:
>>>> On Thu, Apr 14, 2022 at 01:39:23PM -0700, Stefano Stabellini wrote:
>>>>> OK, now we know that the code path with Xen is correct and it is the
>>>>> same code path taken (dma_alloc_direct) as when !CONFIG_XEN and !SMMU.
>>>>> That is how it should be.
>>>>>
>>>>> I cannot explain why dma_alloc_direct() would fail when called from
>>>>> xen_swiotlb_alloc_coherent(), but it would succeed when called from
>>>>> dma_alloc_attrs() without Xen.
>>>>>
>>>>> I am not aware of any restrictions that xen or swiotlb-xen would
>>>>> introduce in that regard. Unless you are just running out of memory
>>>>> because dom0_mem too low.
>>>>
>>>> The crash is deep down in the page allocator.  Even if memory was low
>>>> it should no crash.  So there is some odd interaction between Xen
>>>> and the page allocator going on.  I think nvme and dma-direct really
>>>> are only the messenger here.
>>>
>>>
>>> I cannot think of anything but if that is the case I guess it is more
>>> likely related to reserved-memory not properly advertised or ACPI tables
>>> not properly populated.
>>
>> I am not sure if it is true as we are able to boot with the same reserved memory or
>> the same ACPI table populated if we boot without swiotlb-xen dma ops.
>>
>>>
>>>
>>> Rahul,
>>>
>>> What happens if you boot Linux on Xen with swiotlb-xen disabled?
>>
>> Linux boots fine without any issue if we disable swiotlb-xen as mentioned below.
>
> The plot thinkens.
>
> Without swiotlb-xen, Linux boots fine. With swiotlb-xen it crashes.
> However, in both cases, the very same memory allocation function is
> used: dma_direct_alloc. In one case it works, in the other case it
> crashes.  Everything else is the same.
>
> There are a couple of questionable things with dma masks in
> xen_swiotlb_alloc_coherent, but they are *after* the call to
> xen_alloc_coherent_pages, which is the one that crashes. So they cannot
> be the cause of the crash.
>
> Before the call to xen_alloc_coherent_pages, there is only:
>
>  1) flags &= ~(__GFP_DMA | __GFP_HIGHMEM);
>  2) size = 1UL << (order + XEN_PAGE_SHIFT);
>
>
> 1) is already done by dma_alloc_attrs, so it is superfluous. I couldn't
> explain how 2) could possibly trigger the crash.  XEN_PAGE_SHIFT is
> always 12 even on 64K pages kernels. You can try removing 2) from
> xen_swiotlb_alloc_coherent, but we are really wandering in the dark
> here.

I tried removing the 2) but after that also issue remains.

>
> Then there is xen_swiotlb_init() which allocates some memory for
> swiotlb-xen at boot. It could lower the total amount of memory
> available, but if you disabled swiotlb-xen like I suggested,
> xen_swiotlb_init() still should get called and executed anyway at boot
> (it is called from arch/arm/xen/mm.c:xen_mm_init). So xen_swiotlb_init()
> shouldn't be the one causing problems.
>
> That's it -- there is nothing else in swiotlb-xen that I can think of.
>
> I don't have any good ideas, so I would only suggest to add more printks
> and report the results, for instance:

As suggested I added the more printks but only difference I see is the size apart
from that everything looks same .

Please find the attached logs for xen and native linux boot.

Regards
Rahul

Attachment: xen_boot_with_debug.log
Description: xen_boot_with_debug.log

Attachment: native_linux_boot_debug.log
Description: native_linux_boot_debug.log