[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] PCI passthrough performance loss with Skylake-SP

> -----Original Message-----
> From: Roger Pau Monne
> Sent: 26 June 2018 07:57
> To: Andreas Kinzler <hfp@xxxxxxxxx>
> Cc: xen-devel@xxxxxxxxxxxxx; Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> Subject: Re: [Xen-devel] PCI passthrough performance loss with Skylake-SP
> On Mon, Jun 25, 2018 at 05:32:58PM +0200, Andreas Kinzler wrote:
> > I am currently testing PCI passthrough on the Skylake-SP platform using a
> > Supermicro X11SPi-TF mainboard. Using PCI passthrough (an LSI SAS HBA)
> > causes severe performance loss on the Skylake-SP platform while Xeon E3
> v5
> > is not affected at all. The test uses standard iperf3 as a client - the
> > passed PCI device is not used in the test - so that just the presence of the
> > passed device will cause the iperf3 performance to drop from 6.5 gbit/sec
> > (no passthrough) to 4.5 gbit/sec.
> I assume that the network interface that you are testing is a PV
> network interface?
> > Any explanation/fixes for that?
> Are both systems using the same version of Xen and Linux?
> Paul discovered that (recent?) Linux versions the grant table is
> allocated in a MMIO area, which has the memory cache attribute set to
> uncached. This has a noticeable impact on performance when the guest
> also has a passthrough device.

I can't necessarily claim credit for the discovery but that is indeed the case, 
and the sort of performance drop seen is exactly what I'd expect. I recently 
put a change into the Windows PV drivers to use a ballooned-out region of the 
guest RAM to host the grant tables instead, which avoids this problem.

> Note that this is not an issue without passthrough because Xen forces
> the cache attributes of memory to WB in that case, regardless of the
> guest choice.

We run with this little hack in XenServer, which also 'fixes' things for guests 
OS that have not been modified:

diff --git a/xen/arch/x86/hvm/mtrr.c b/xen/arch/x86/hvm/mtrr.c
index 3c51244..d0126b4 100644
--- a/xen/arch/x86/hvm/mtrr.c
+++ b/xen/arch/x86/hvm/mtrr.c
@@ -811,7 +811,8 @@ int epte_get_entry_emt(struct domain *d, unsigned long gfn, 
mfn_t mfn,
         return MTRR_TYPE_UNCACHABLE;
-    if ( !need_iommu(d) && !cache_flush_permitted(d) )
+    if ( (!need_iommu(d) && !cache_flush_permitted(d)) ||
+         is_xen_heap_mfn(mfn_x(mfn)) )
         *ipat = 1;
         return MTRR_TYPE_WRBACK;

So you may want to give that a try.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.