[Xen-devel] Intel GPU pass-through with > 3G

[Jean Guyader <jean.guyader@xxxxxxxxx>]

Hello,

I'm passing through a graphic card to a guest that has more than 3G of
RAM (4G to be precise in my case).

What happen is that the VM creation is stuck in the process, so I put
some tracing in the Xen code to see what
was taking the time. I discovered that the guest was stuck in
hvmloader inside this loop:

   while ( (pci_mem_start >> PAGE_SHIFT) < hvm_info->low_mem_pgend )
    {
        struct xen_add_to_physmap xatp;
        if ( hvm_info->high_mem_pgend == 0 )
            hvm_info->high_mem_pgend = 1ull << (32 - PAGE_SHIFT);
        xatp.domid = DOMID_SELF;
        xatp.space = XENMAPSPACE_gmfn;
        xatp.idx   = --hvm_info->low_mem_pgend;
        xatp.gpfn  = hvm_info->high_mem_pgend++;
        if ( hypercall_memory_op(XENMEM_add_to_physmap, &xatp) != 0 )
            BUG();
    }

This loop relocate the RAM on the top to leave so space for the PCI BARs.
It's loop on each page so in my case it's quite a big loop because the
GPU has a BAR of 256M.

So the interesting is that the function add_to_physmap takes most of
the time. I believe
that what takes most part of it is the iommu iotlb flush that come
with the iommu_map_pages
or the iommu_unmap_page which are called when we manipulate the p2m table.

In my case the iommu flush take a very long time (because of the intel
gpu ?), about 10
milliseconds. So if I'm patient enough my domain will start, about 10 minutes.

A way to go will be to create a range interface to iommu_map_page
iommu_unmap_page
since iommu_flush are so expensive. Then some work need to be done to
add a range interface
to all the function between add_to_physmap and the p2m_set_entry which
would be a big
patch. I hope there is another way out of this problem.

Thanks,
Jean

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel