[Xen-devel] Shattering superpages impact on IOMMU in Xen

[Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>]

Hi, all.

Playing with non-shared IOMMU in Xen on ARM I faced one interesting
thing. I found out that the superpages were shattered during domain
life cycle.
This is the result of mapping of foreign pages, ballooning memory,
even if domain maps Xen shared pages, etc.
I don't bother with the memory fragmentation at the moment. But,
shattering bothers me from the IOMMU point of view.
As the Xen owns IOMMU it might manipulate IOMMU page tables when
passthoughed/protected device doing DMA in Linux. It is hard to detect
when the DMA transaction isn't in progress
in order to prevent this race. So, if we have inflight transaction
from a device when changing IOMMU mapping we might get into trouble.
Unfortunately, not in all the cases the
faulting transaction can be restarted. The chance to hit the problem
increases during shattering.

I did next test:
The dom0 on my setup contains ethernet IP that are protected by IOMMU.
What is more, as the IOMMU I am playing with supports superpages (2M,
1G) the IOMMU driver
takes into account these capabilities when building page tables. As I
gave 256 MB for dom0, the IOMMU mapping was built by 2M memory blocks
only. As I am using NFS for both dom0 and domU the ethernet IP
performs DMA transactions almost all the time.
Sometimes, I see the IOMMU page faults during creating guest domain. I
think, it happens during Xen is shattering 2M mappings 4K mappings (it
unmaps dom0 pages by one 4K page at a time, then maps domU pages there
for copying domU images).
But, I don't see any page faults when the IOMMU page table was built
by 4K pages only.

I had a talk with Julien on IIRC and we came to conclusion that the
safest way would be to use 4K pages to prevent shattering, so the
IOMMU shouldn't report superpage capability.
On the other hand, if we build IOMMU from 4K pages we will have
performance drop (during building, walking page tables), TLB pressure,
etc.
Another possible solution Julien was suggesting is to always
ballooning with 2M, 1G, and not using 4K. That would help us to
prevent shattering effect.
The discussion was moved to the ML since it seems to be a generic
issue and the right solution should be think of.

What do you think is the right way to follow? Use 4K pages and don't
bother with shattering or try to optimize? And if the idea to make
balloon mechanism smarter makes sense how to teach balloon to do so?
Thank you.

-- 
Regards,

Oleksandr Tyshchenko

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel