Re: [Xen-devel] Stop the continuous flood of (XEN) traps.c:2432:d0 Domain attempted WRMSR ..

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Wed, Mar 28, 2012 at 04:29:07PM -0400, Konrad Rzeszutek Wilk wrote:
> On Thu, Feb 09, 2012 at 01:27:15PM -0800, Jesse Barnes wrote:
> > On Thu, 9 Feb 2012 17:21:47 -0400
> > Konrad Rzeszutek Wilk <konrad@xxxxxxxxxx> wrote:
> > 
> > > On Sun, Feb 05, 2012 at 09:44:13PM +0200, Pasi K?rkk?inen wrote:
> > > > On Fri, Feb 03, 2012 at 01:55:27PM -0500, Konrad Rzeszutek Wilk wrote:
> > > > > On Fri, Feb 03, 2012 at 08:09:52PM +0200, Pasi K?rkk?inen wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > IIRC there was some discussion earlier about these messages in 
> > > > > > Xen's dmesg:
> > > > > > 
> > > > > > (XEN) traps.c:2432:d0 Domain attempted WRMSR 00000000000001ac from 
> > > > > > 0x0000000000c800c8 to 0x0000000080c880c8.
> > > > > > (XEN) traps.c:2432:d0 Domain attempted WRMSR 00000000000001ac from 
> > > > > > 0x0000000000c800c8 to 0x0000000080c880c8.
> > > > > > (XEN) traps.c:2432:d0 Domain attempted WRMSR 00000000000001ac from 
> > > > > > 0x0000000000c800c8 to 0x0000000080c880c8.
> > > > > > (XEN) traps.c:2432:d0 Domain attempted WRMSR 00000000000001ac from 
> > > > > > 0x0000000000c800c8 to 0x0000000080c880c8.
> > > > > > 
> > > > > > At least on my systems there's continuous flood of those messages, 
> > > > > > so they will fill up the
> > > > > > Xen dmesg log buffer and "xm dmesg" or "xl dmesg" won't show any 
> > > > > > valuable information, just those messages.
> > > > > 
> > > > > Is it always that MSR? That looks to be TURBO_POWER_CURRENT_LIMIT
> > > > > which is the intel_ips driver doing.
> > > > > 
> > > > 
> > > > Yeah, it's always the same..
> > > > 
> > > > > > 
> > > > > > I seem to be getting those messages even when there's only dom0 
> > > > > > running.
> > > > > > Is the plan to drop those messages? What's causing them? 
> > > > > 
> > > > > Looks to be the intel-ips. If you rename it does the issue disappear?
> > > > 
> > > > I just did "rmmod intel_ips" and the flood stopped.. 
> > > > 
> > > > 
> > > > Btw on baremetal I get this in dmesg:
> > > > 
> > > > [  745.033645] CPU1: Core temperature above threshold, cpu clock 
> > > > throttled (total events = 1)
> > > > [  745.033652] CPU3: Core temperature above threshold, cpu clock 
> > > > throttled (total events = 1)
> > > > [  745.034676] CPU1: Core temperature/speed normal
> > > > [  745.034678] CPU3: Core temperature/speed normal
> > > > [  849.678508] intel ips 0000:00:1f.6: MCP limit exceeded: Avg temp 
> > > > 9682, limit 9000
> > > > [  899.614074] intel ips 0000:00:1f.6: MCP limit exceeded: Avg temp 
> > > > 9896, limit 9000
> > > > [  899.722881] [Hardware Error]: Machine check events logged
> > > > [ 1172.675987] CPU3: Core temperature above threshold, cpu clock 
> > > > throttled (total events = 78)
> > > > [ 1172.675990] CPU1: Core temperature above threshold, cpu clock 
> > > > throttled (total events = 78)
> > > > [ 1172.677038] CPU1: Core temperature/speed normal
> > > > [ 1172.677042] CPU3: Core temperature/speed normal
> > > > [ 1174.260050] intel ips 0000:00:1f.6: MCP limit exceeded: Avg temp 
> > > > 9676, limit 9000
> > > > [ 1199.339634] [Hardware Error]: Machine check events logged
> > > 
> > > Jesse, and Matthew,
> > > 
> > > Is there a way to make the intel_ips.c driver be in a "low-power" state?
> > > 
> > > My first thought about fixing this was that we could allow the
> > > hypervisor to allow those RDMSR but the Linux kernel has no power to
> > > actually influence the power management (as the hypervisor is in charge
> > > of that) - so would the driver be capable of just sitting back and
> > > not influencing the CPU?
> > 
> > Yeah it's easy enough to turn off or disable.  But it doesn't currently
> > export any knobs for controlling behavior.  I don't have any issue with
> > exposing some though...
> 
> Pasi,
> 
> Could you test the two patches independetly of each other? Meaning
> test the Linux one without the Xen one, and vice-versa.
> 
> 

Sorry for the really long delay.. I tested these patches now.


> diff --git a/drivers/platform/x86/intel_ips.c 
> b/drivers/platform/x86/intel_ips.c
> index 88a98cf..7276831 100644
> --- a/drivers/platform/x86/intel_ips.c
> +++ b/drivers/platform/x86/intel_ips.c
> @@ -1407,6 +1407,10 @@ static struct ips_mcp_limits *ips_detect_cpu(struct 
> ips_driver *ips)
>       }
>  
>       rdmsrl(TURBO_POWER_CURRENT_LIMIT, turbo_power);
> +     if (turbo_power == 0) {
> +             ips->turbo_toggle_allowed = false;
> +             return NULL;
> +     }
>       tdp = turbo_power & TURBO_TDP_MASK;
>  
>       /* Sanity check TDP against CPU */

This Linux patch applied to Linux 3.4.4 dom0 kernel and no patches to the 
hypervisor
didn't change anything.. the hypervisor log is still flooded with:

(XEN) traps.c:2432:d0 Domain attempted WRMSR 00000000000001ac from 
0x0000000000c800c8 to 0x0000000080c880c8.
(XEN) traps.c:2432:d0 Domain attempted WRMSR 00000000000001ac from 
0x0000000000c800c8 to 0x0000000080c880c8.


And then the Xen patch..

> diff -r 8e2690dbec49 xen/arch/x86/traps.c
> --- a/xen/arch/x86/traps.c    Sat Mar 24 13:13:49 2012 -0400
> +++ b/xen/arch/x86/traps.c    Wed Mar 28 16:27:31 2012 -0400
> @@ -1746,7 +1746,8 @@ void (*pv_post_outb_hook)(unsigned int p
>  static inline uint64_t guest_misc_enable(uint64_t val)
>  {
>      val &= ~(MSR_IA32_MISC_ENABLE_PERF_AVAIL |
> -             MSR_IA32_MISC_ENABLE_MONITOR_ENABLE);
> +             MSR_IA32_MISC_ENABLE_MONITOR_ENABLE |
> +             MSR_IA32_MISC_ENABLE_TURBO);
>      val |= MSR_IA32_MISC_ENABLE_BTS_UNAVAIL |
>             MSR_IA32_MISC_ENABLE_PEBS_UNAVAIL |
>             MSR_IA32_MISC_ENABLE_XTPR_DISABLE;
> diff -r 8e2690dbec49 xen/include/asm-x86/msr-index.h
> --- a/xen/include/asm-x86/msr-index.h Sat Mar 24 13:13:49 2012 -0400
> +++ b/xen/include/asm-x86/msr-index.h Wed Mar 28 16:27:31 2012 -0400
> @@ -327,6 +327,7 @@
>  #define MSR_IA32_MISC_ENABLE_MONITOR_ENABLE (1<<18)
>  #define MSR_IA32_MISC_ENABLE_LIMIT_CPUID  (1<<22)
>  #define MSR_IA32_MISC_ENABLE_XTPR_DISABLE (1<<23)
> +#define MSR_IA32_MISC_ENABLE_TURBO        (1<<38)
>  
>  #define MSR_IA32_TSC_DEADLINE                0x000006E0
>  #define MSR_IA32_ENERGY_PERF_BIAS    0x000001b0

It seems this Xen patch breaks compilation.. at least on Fedora 16 gcc (Xen 
4.1.3-rc2):

traps.c: In function 'guest_misc_enable':
traps.c:1780:14: error: left shift count >= width of type [-Werror]
cc1: all warnings being treated as errors
make[4]: *** [traps.o] Error 1

So I had to do a trivial change to msr-index.h:
#define MSR_IA32_MISC_ENABLE_TURBO        (1L<<38)

Which seems to fix the compilation. 

I tested the patched hypervisor with stock Fedora 16 Linux 3.4.2-1.fc16.x86_64 
dom0 kernel,
and also then I get the hypervisor log entries:

(XEN) traps.c:2489:d0 Domain attempted WRMSR 00000000000001ac from 
0x0000000000c800c8 to 0x0000000080c880c8.
(XEN) traps.c:2489:d0 Domain attempted WRMSR 00000000000001ac from 
0x0000000000c800c8 to 0x0000000080c880c8.

So it looks like unfortunately the patches didn't help reducing the spam in the 
hypervisor logs.

Thanks,

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel