Re: [PATCH v2] x86/intel: ensure Global Performance Counter Control is setup correctly

[Jan Beulich <jbeulich@xxxxxxxx>]

On 11.01.2024 10:08, Roger Pau Monne wrote:
> When Architectural Performance Monitoring is available, the PERF_GLOBAL_CTRL
> MSR contains per-counter enable bits that is ANDed with the enable bit in the
> counter EVNTSEL MSR in order for a PMC counter to be enabled.
> 
> So far the watchdog code seems to have relied on the PERF_GLOBAL_CTRL enable
> bits being set by default, but at least on some Intel Sapphire and Emerald
> Rapids this is no longer the case, and Xen reports:
> 
> Testing NMI watchdog on all CPUs: 0 40 stuck
> 
> The first CPU on each package is started with PERF_GLOBAL_CTRL zeroed, so PMC0
> doesn't start counting when the enable bit in EVNTSEL0 is set, due to the
> relevant enable bit in PERF_GLOBAL_CTRL not being set.
> 
> Check and adjust PERF_GLOBAL_CTRL during CPU initialization so that all the
> general-purpose PMCs are enabled.  Doing so brings the state of the 
> package-BSP
> PERF_GLOBAL_CTRL in line with the rest of the CPUs on the system.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> ---
> Changes since v1:
>  - Do the adjustment of PERF_GLOBAL_CTRL even if the watchdog is not used, and
>    enable all counters.
> ---
> Unsure whether printing a warning if the value of PERF_GLOBAL_CTRL is not
> correct is of any value, hence I haven't added it.
> ---
>  xen/arch/x86/cpu/intel.c | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/x86/cpu/intel.c b/xen/arch/x86/cpu/intel.c
> index dfee64689ffe..40d3eb0e18a7 100644
> --- a/xen/arch/x86/cpu/intel.c
> +++ b/xen/arch/x86/cpu/intel.c
> @@ -533,9 +533,25 @@ static void cf_check init_intel(struct cpuinfo_x86 *c)
>       init_intel_cacheinfo(c);
>       if (c->cpuid_level > 9) {
>               unsigned eax = cpuid_eax(10);
> +             unsigned int cnt = (uint8_t)(eax >> 8);
> +
>               /* Check for version and the number of counters */
> -             if ((eax & 0xff) && (((eax>>8) & 0xff) > 1))
> +             if ((eax & 0xff) && (cnt > 1) && (cnt <= 32)) {

I may not have looked closely enough, but I didn't find the limit of
32 being stated anywhere.

> +                     uint64_t global_ctrl;
> +                     unsigned int cnt_mask = (1UL << cnt) - 1;

Bits 2 + 4 * n have an additional qualification as per SDM vol 4.

Jan

> +                     /*
> +                      * On (some?) Sapphire/Emerald Rapids platforms each
> +                      * package-BSP starts with all the enable bits for the
> +                      * general-purpose PMCs cleared.  Adjust so counters
> +                      * can be enabled from EVNTSEL.
> +                      */
> +                     rdmsrl(MSR_CORE_PERF_GLOBAL_CTRL, global_ctrl);
> +                     if ((global_ctrl & cnt_mask) != cnt_mask)
> +                             wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL,
> +                                    global_ctrl | cnt_mask);
>                       __set_bit(X86_FEATURE_ARCH_PERFMON, c->x86_capability);
> +             }
>       }
>  
>       if ( !cpu_has(c, X86_FEATURE_XTOPOLOGY) )