Re: [RFC] Avoid dom0/HVM performance penalty from MSR access tightening

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, Feb 10, 2022 at 11:27:15AM -0600, Alex Olson wrote:
> I'm seeing strange performance issues under Xen on a Supermicro server with a 
> Xeon D-1541 CPU caused by an MSR-related commit.
> 
> Commit 322ec7c89f6640ee2a99d1040b6f786cf04872cf 'x86/pv: disallow access to 
> unknown MSRs'
> surprisingly introduces a severe performance penality where dom0 has about 
> 1/8th
> the normal CPU performance. Even even when 'xenpm' is used to select the
> performance governor and operate the CPU at maximum frequency, actual CPU
> performance is still 1/2 of normal (as well as using 
> "cpufreq=xen,performance").
> 
> The patch below fixes it but I don't fully understand why.
> 
> Basically, when *reads* of MSR_IA32_THERM_CONTROL are blocked, dom0 and
> guests (pinned to other CPUs) see the performance issues.

You only mention MSR_IA32_THERM_CONTROL here...

> For benchmarking purposes, I built a small C program that runs a "for
> loop" 
> 4Billion iterations and timed its execution. In dom0, the
> performance issues
> also cause HVM guest startup time to go from 9-10
> seconds to almost 80 seconds.
> 
> I assumed Xen was managing CPU frequency and thus blocking related MSR
> access by dom0 (or any other domain). However,  clearly something else
> is happening and I don't understand why.
> 
> I initially attempted to copy the same logic as the write MSR case. This
> was effective at fixing the dom0 performance issue, but still left other
> domains running at 1/2 speed. Hence, the change below has no access control.
> 
> 
> If anyone has any insight as to what is really happening, I would be all ears
> as I am unsure if the change below is a proper solution.
> 
> Thanks
> 
> -Alex
> 
> ---
> ---
>  xen/arch/x86/pv/emul-priv-op.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/xen/arch/x86/pv/emul-priv-op.c b/xen/arch/x86/pv/emul-priv-op.c
> index 7f4279a051..f254479bda 100644
> --- a/xen/arch/x86/pv/emul-priv-op.c
> +++ b/xen/arch/x86/pv/emul-priv-op.c
> @@ -970,6 +970,18 @@ static int read_msr(unsigned int reg, uint64_t *val,
>          *val = 0;
>          return X86EMUL_OKAY;
>  
> +    /* being unable to read MSR_IA32_THERM_CONTROL seems to significantly 
> affect
> +     * dom0 and thus HVM guest startup performance, as well as PVH VMs.
> +     */
> +    case MSR_IA32_THERM_CONTROL:
> +    case MSR_IA32_ENERGY_PERF_BIAS:

...yet in the patch you also allow access to
MSR_IA32_ENERGY_PERF_BIAS, which makes me wonder whether
MSR_IA32_THERM_CONTROL is the only required one.

It could help to post full logs Xen + Linux dmesgs.

Is this reproducible with different Linux versions?

Thanks, Roger.