Re: [Xen-devel] [PATCH v3 7/7] x86/tlb: use Xen L0 assisted TLB flush when available

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, Feb 06, 2020 at 03:46:56PM +0100, Jan Beulich wrote:
> On 06.02.2020 15:09, Roger Pau Monné wrote:
> > On Thu, Feb 06, 2020 at 01:49:35PM +0000, Wei Liu wrote:
> >> On Mon, Jan 27, 2020 at 07:11:15PM +0100, Roger Pau Monne wrote:
> >>> Use Xen's L0 HVMOP_flush_tlbs hypercall in order to perform flushes.
> >>> This greatly increases the performance of TLB flushes when running
> >>> with a high amount of vCPUs as a Xen guest, and is specially important
> >>> when running in shim mode.
> >>>
> >>> The following figures are from a PV guest running `make -j32 xen` in
> >>> shim mode with 32 vCPUs and HAP.
> >>>
> >>> Using x2APIC and ALLBUT shorthand:
> >>> real      4m35.973s
> >>> user      4m35.110s
> >>> sys       36m24.117s
> >>>
> >>> Using L0 assisted flush:
> >>> real    1m2.596s
> >>> user    4m34.818s
> >>> sys     5m16.374s
> >>>
> >>> The implementation adds a new hook to hypervisor_ops so other
> >>> enlightenments can also implement such assisted flush just by filling
> >>> the hook. Note that the Xen implementation completely ignores the
> >>> dirty CPU mask and the linear address passed in, and always performs a
> >>> global TLB flush on all vCPUs.
> >>>
> >>> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> >>> ---
> >>> Changes since v1:
> >>>  - Add a L0 assisted hook to hypervisor ops.
> >>> ---
> >>>  xen/arch/x86/guest/hypervisor.c        | 10 ++++++++++
> >>>  xen/arch/x86/guest/xen/xen.c           |  6 ++++++
> >>>  xen/arch/x86/smp.c                     | 11 +++++++++++
> >>>  xen/include/asm-x86/guest/hypervisor.h | 17 +++++++++++++++++
> >>>  4 files changed, 44 insertions(+)
> >>>
> >>> diff --git a/xen/arch/x86/guest/hypervisor.c 
> >>> b/xen/arch/x86/guest/hypervisor.c
> >>> index 4f27b98740..4085b19734 100644
> >>> --- a/xen/arch/x86/guest/hypervisor.c
> >>> +++ b/xen/arch/x86/guest/hypervisor.c
> >>> @@ -18,6 +18,7 @@
> >>>   *
> >>>   * Copyright (c) 2019 Microsoft.
> >>>   */
> >>> +#include <xen/cpumask.h>
> >>>  #include <xen/init.h>
> >>>  #include <xen/types.h>
> >>>  
> >>> @@ -64,6 +65,15 @@ void hypervisor_resume(void)
> >>>          ops->resume();
> >>>  }
> >>>  
> >>> +int hypervisor_flush_tlb(const cpumask_t *mask, const void *va,
> >>> +                         unsigned int order)
> >>> +{
> >>> +    if ( ops && ops->flush_tlb )
> >>> +        return ops->flush_tlb(mask, va, order);
> >>> +
> >>
> >> Is there a way to make this an alternative call? I consider tlb flush a
> >> frequent operation which can use some optimisation.
> >>
> >> This can be done as a later improvement if it is too difficult though.
> >> This patch already has some substantial improvement.
> > 
> > I can look into making this an alternative call, if it turn out to be
> > too complex I will leave it out for a separate patch.
> 
> It'll be two steps - make a global struct hypervisor_ops instance
> which the per-hypervisor instances get _copied_ into upon boot
> (at that point all of those can go into .init.* sections), and
> then switch the call(s) of interest. I.e. while the 2nd step can
> of course be done right here, the first will want to be in a
> prereq patch.

Done. I've only switched the flush_tlb operation, since the rest are
not relevant from a performance PoV.

Will wait until next week before posting a new version.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel