Re: Disable swiotlb for Dom0

[Roman Skakun <Roman_Skakun@xxxxxxxx>]

>> But I think, we cannot use64af97000 address in the swiotlb_bounce()

>> directly.
>>
>I am not sure to understand what you mean. Can you clarify?

I thought, that the address 64af97000 can be used directly here:
https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L572
and I was confused about it.

After some diggings I realized that this code is not reachable because 
DMA_ATTR_SKIP_CPU_SYNC is active.

>> Swiotlb did not fit requested slots because the maximum slot size equals
>> IO_TLB_SEGSIZE=128 by default.
>
>> Ok. So it sounds like your problem is the have a DMA buffer that is too
>> large for the default swiotlb. Did you try to bump the value from the
>> command line?

Yes, you are right. I checked the implementation more detailed and figured out that
swiotlb tries to find a contiguous segment that is a big enough, as shown here:
https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528,

But the maximum TLB segment equals 128K by default in accordance with
https://elixir.bootlin.com/linux/v5.10/source/include/linux/swiotlb.h#L25

This means that the max size of TLB segment equals:
128 * 2048 = 256K
 
After this, I tried to increase the TLB segment size like this:

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h

index fbdc65782195..85f32865bb6c 100644

--- a/include/linux/swiotlb.h

+++ b/include/linux/swiotlb.h

@@ -22,7 +22,7 @@ enum swiotlb_force {

  * must be a power of 2.  What is the appropriate value ?

  * The complexity of {map,unmap}_single is linearly dependent on this value.

  */

-#define IO_TLB_SEGSIZE 128

+#define IO_TLB_SEGSIZE 2048

 

 /*

  * log of the size of each IO TLB slab.  The number of slabs is command line

and the problem is gone.

Also, I found the article https://www.xilinx.com/support/answers/72694.html, 
where I believe the same issue was mentioned.

Thank you so much for your time and help!

---

From: Julien Grall <julien@xxxxxxx>
Sent: Friday, August 13, 2021 1:51 PM
To: Roman Skakun <Roman_Skakun@xxxxxxxx>; sstabellini@xxxxxxxxxx <sstabellini@xxxxxxxxxx>
Cc: Bertrand Marquis <bertrand.marquis@xxxxxxx>; Andrii Anisov <Andrii_Anisov@xxxxxxxx>; Volodymyr Babchuk <Volodymyr_Babchuk@xxxxxxxx>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx <xen-devel@xxxxxxxxxxxxxxxxxxxx>; Roman Skakun <rm.skakun@xxxxxxxxx>; Jan Beulich <jbeulich@xxxxxxxx>
Subject: Re: Disable swiotlb for Dom0

 

On 13/08/2021 10:38, Roman Skakun wrote:
> Hi Julien,

Hi Roman,

>> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.
>>
>> swiotlb-xen on Arm will convert it to the MFN because it is not aware
>> whether the device is behind an IOMMU.
>
> If I'm understand right, it seems like that swiotlb-xen is not ready to
> work properly in case
> when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
> Maybe you know some ideas to overcome this condition?

swiotlb-xen work as intended. You have a DMA buffer at an address that
your device cannot deal with. So it will try to bounce it.

>
>>  As the address is too high to be handled by the device, swiotlb will try
>>  to bounce it. I think it is correct to bounce the page but I am not sure
>>  why it can't. What the size of the DMA transaction?
>
> The DMA map size is 3686400 bytes.

So that's a 3MB buffer. I am slightly confused because in an earlier
message you wrote that the memory is coming from the guest. How did you
map it?

>
> I've added several logs to swiotlb map_single() and see:
> [  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr:
> 64af97000, needed: 708,
> avail: 7fc0, stride: 2, index: 4160

I am not sure how to read the logs... Are the number in hexadecimal or
decimal? It might be useful if you post the diff of your changes.

[...]

> Swiotlb did not fit requested slots because the maximum slot size equals
> IO_TLB_SEGSIZE=128 by default.

Ok. So it sounds like your problem is the have a DMA buffer that is too
large for the default swiotlb. Did you try to bump the value from the
command line?

> But I think, we cannot use64af97000 address in the swiotlb_bounce()
> directly.

I am not sure to understand what you mean. Can you clarify?

Cheers,

[...]

--
Julien Grall