|
Hi Haitao,
Am 20.11.2010 schrieb ""Shan, Haitao" <haitao.shan@xxxxxxxxx>":
> Hi, Jan,
>
> The actual handler core2_vpmu_do_interrupt is never called under a NMI handler context. The reason is that I don't want to place such an interrupt on behalf of guest to be of so high a priority. So, even guest programs the virtual APIC to use NMI, the underlying HW does not. It uses a vector (0xF8? I don't remember clearly).
>
> This issue this patch solves is likely a HW bug but I have not find any documented errata. On some NHM processors, when a PMI is received, you will observe the counter that triggers this interrupt is zero (which means it has just gone overflowed). If you unmask the PMI sources (as you might know, PMI gets automatically masked when received), you will immediately get another interrupt. This is how you get an interrupt loop here (not NMI loop, but PMI loop).
> The reason why native oprofile works is that oprofile actually reprogram the counter before unmask the interrupt. In our VPMU implementation in Xen, the PMI handler does nothing but pending an interrupt to guest, then it unmasks the PMI and return.
> I find that reprograming the counter to be 1 works around this issue (any number that is *not* zero should work actually).
> Another working around would be only unmasking the real PMI when guests unmask its virtual PMIs. This work around is not very promising, since by the time guests unmask the virtual PMIs, the vcpu might be migrated to other CPUs. You need complex tracking and an IPI to do the right unmask on the correct physical CPUs.
>
> Hope I can explain the whole matter clearly here.
>
>
> BTW: As I observed on other processors, when a PMI is received, the counter is never zero (some values just slightly greater than zero).
>
> Shan Haitao
very good explanation, many thanks!
Dietmar.
>
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@xxxxxxxxxx]
> Sent: Friday, November 19, 2010 7:07 PM
> To: Dietmar Hahn
> Cc: Shan, Haitao; xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Xen-devel] [PATCH] VPMU issue on Nehalem cpus
>
> >>> On 19.11.10 at 11:29, Dietmar Hahn <dietmar.hahn@xxxxxxxxxxxxxx> wrote:
> > +/*
> > + * QUIRK to workaround an issue on Nehalem processors currently seen
> > + * on family 6 cpus E5520 (model 26) and X7542 (model 46).
> > + * The issue leads to endless NMI loops on the processor.
> > + * If a counter triggers an NMI and while the NMI handler is running another
> > + * counter overflows the second counter triggers endless new NMIs.
> > + * A solution is to read all flagged counters and if the value is 0 write
> > + * 1 into it.
> > + */
>
> Two things I don't understand here: One is that I can't see how
> from the NMI handler control would get to
> core2_vpmu_do_interrupt() - afaics, this gets called only in the
> context of the (vectored) smp_pmu_apic_interrupt(). The other
> is that if nested interrupts occur, how would you prevent this
> by writing ones into zero counters? That is, in the best case I
> could see this shrinking the window within which unintended
> nested interrupts would occur. Or is it that the secondary
> interrupts only occur after the first one returned? Is this (mis-)
> behavior documented somewhere?
>
> > +static int is_nmi_quirk;
>
> bool_t __read_mostly?
>
> > +
> > +static void check_nmi_quirk(void)
> > +{
> > + u8 family = current_cpu_data.x86;
> > + u8 cpu_model = current_cpu_data.x86_model;
> > + is_nmi_quirk = 0;
> > + if ( family == 6 )
> > + {
> > + if ( cpu_model == 46 || cpu_model == 26 )
> > + is_nmi_quirk = 1;
> > + }
> > +}
> > +
> > +static int core2_get_pmc_count(void);
> > +static void handle_nmi_quirk(u64 msr_content)
> > +{
> > + int num_gen_pmc = core2_get_pmc_count();
> > + int num_fix_pmc = 3;
> > + int i;
> > + u64 val;
> > +
> > + if ( !is_nmi_quirk )
> > + return;
> > +
> > + val = msr_content & ((1 << num_gen_pmc) - 1);
>
> What's the point of masking if the subsequent loop looks at the
> bottom so many bits only anyway?
>
> > + for ( i = 0; i < num_gen_pmc; i++ )
> > + {
> > + if ( val & 0x1 )
> > + {
> > + u64 cnt;
> > + rdmsrl(MSR_P6_PERFCTR0 + i, cnt);
> > + if ( cnt == 0 )
> > + wrmsrl(MSR_P6_PERFCTR0 + i, 1);
> > + }
> > + val >>= 1;
> > + }
> > + val = (msr_content >> 32) & ((1 << num_fix_pmc) - 1);
>
> Same here.
>
> > + for ( i = 0; i < num_fix_pmc; i++ )
> > + {
> > + if ( val & 0x1 )
> > + {
> > + u64 cnt;
> > + rdmsrl(MSR_CORE_PERF_FIXED_CTR0 + i, cnt);
> > + if ( cnt == 0 )
> > + wrmsrl(MSR_CORE_PERF_FIXED_CTR0 + i, 1);
> > + }
> > + val >>= 1;
> > + }
> > +}
> > +
> > +#define CHECK_HANDLE_NMI_QUIRK(msr_content) \
> > + if ( is_nmi_quirk ) \
> > + handle_nmi_quirk(msr_content);
> > +
>
> Why do you need a macro here if you use it only once?
>
> > u32 core2_counters_msr[] = {
> > MSR_CORE_PERF_FIXED_CTR0,
> > MSR_CORE_PERF_FIXED_CTR1,
> > @@ -494,6 +558,9 @@
> > rdmsrl(MSR_CORE_PERF_GLOBAL_STATUS, msr_content);
> > if ( !msr_content )
> > return 0;
> > +
> > + CHECK_HANDLE_NMI_QUIRK(msr_content)
> > +
> > core2_vpmu_cxt->global_ovf_status |= msr_content;
> > msr_content = 0xC000000700000000 | ((1 << core2_get_pmc_count()) - 1);
> > wrmsrl(MSR_CORE_PERF_GLOBAL_OVF_CTRL, msr_content);
>
> Jan
>
>
>
--
Dietmar Hahn
TSP ES&S SWE OS
FUJITSU
Fujitsu Technology Solutions
Domagkstraße 28, D-80807 München, Germany
Tel: +49 (89) 3222 2952
Email: dietmar.hahn@xxxxxxxxxxxxxx
Web: http://ts.fujitsu.com
Company details: http://ts.fujitsu.com/imprint.html
| 
