RE: [Xen-devel] Potential side-effects and mitigations after cpuidle enabled by default

["Wei, Gang" <gang.wei@xxxxxxxxx>]

On Tuesday, March 31, 2009 10:52 PM, Carsten Schiers wrote:
> Sorry for my ignorance, but as I find all that very interesting, prior to read
> it all over and as I suffer from skew after using cpuidle and also cpufreq
> management (on an AMD CPU ;-, which TSC frequency is variant across
> freq/voltage scaling) a few 
> questions:
> 
>   - you mention lost ticks in some guests, does this include Dom0? It's where
>     my messages mainly show up.

I haven't observe lost ticks warning in Dom0 for current Xen3.4 tip by far.

>   - you recomend to limit cpuidle either to C1 or C2 (in case APIC
>     timer is not stopping. How to know that?

You may need refer to processor's spec.
 
>   - xm debug-key c reports active C1, max_cstate C2, but only lists C1 usage.
>     C1 Clock Ramping seems to be disabled. Platform timer is 25MHz HPET.
>     Excuse my ignorance again, but doesn't that mean I am not using C-states
> at all? 

In Xen3.3 the C1 residency is not counted yet. The max_cstate=C2 does not mean
your platform support C2, it just means if your platform support C-states deeper
than C2, the deepest used C-state will be C2. I guess xm debug-key c didn't 
report
any C2 information (usage, residency) in your platform, right? If yes, that 
means
your system only support C1.

> I understand you speak about Xen 3.4. Currently, I am at 3.3.1 and have to
> wait 
> for a slot to test 3.4. I am curious to see what happens. Dan told me how to
> use 
> xm debug-key t and said, max cycles skew is so much smaller than max stime
> (xen 
> system time) skew. This makes him believe 3.4 will help.

Yes, I also strongly suggesting you to have a try on 3.4. But I doesn't expect 
much
for the variant TSC case, just like what I said in the orginal mail. 

BTW, I believe enabling cpuidle or not should have no impact on your case. Have 
you checked the result while cpufreq disabled?

Thanks
Jimmy

> 
> BR,
> Carsten.
> 
> ----- Originalnachricht -----
> Von: "Wei, Gang" <gang.wei@xxxxxxxxx>
> Gesendet: Die, 31.3.2009 16:00
> An: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>
> Cc: "Tian, Kevin" <kevin.tian@xxxxxxxxx> ; Keir Fraser
> <keir.fraser@xxxxxxxxxxxxx> ; "Yu, Ke" <ke.yu@xxxxxxxxx> Betreff: [Xen-devel]
> Potential side-effects and mitigations after cpuidle enabled by default 
> 
> In xen3.4, cpuidle is defaultly enabled by c/s 19421. But some side-effects
> may exist under different h/w C-states implementations or h/w configurations,
> so that user may occasionally observe latency or system time/tsc skew. Below
> are conditions causing these side-effects and means to mitigate them:
> 
> 1. Latency
> 
> Latency could be caused by two factors: C-state entry/exit latency, and extra
> latency caused by broadcast mechanism.
> 
> C-state entry/exit latency is inevitable since powering on/off gates takes
> time. Normally shallower C-state incurs lighter latency but less power saving
> capability, and vice versa for deeper C-state. Cpuidle governor tries to
> balance performance and power tradeoff in high level, which is one area where
> we'll continue to tune.
> 
> Broadcast is necessary to handle APIC timer stop at deep C-state (>=C3) on
> some platforms. One platform timer source is chosen to carry per-cpu timer
> deadline, and then wakeup CPUs in deep C-state timely at expected expiry.
> By far Xen3.4 supports PIT/HPET as the broadcast source. In current
> implementation PIT broadcast is implemented in periodical mode (10ms) which
> means up to 10ms extra latency could be added on expiry expected from sleep
> CPUs. This is just initial implementation choice which of course could be
> enhanced to on-demand on/off mode in the future. We didn't go into that
> complexity in current implementation, due to its slow access and also short
> wrap count. So HPET broadcast is always preferred, once this facility is
> available which adds negligible overhead with timely wakeup. Then... world is
> not always perfect, and some side-effects also exist along with HPET.
> 
> Detail is listed as below:
> 
> 1.1. For h/w supporting ACPI C1 (halt) only (BIOS reported in ACPI _CST
> method):
> 
> It's immune from this side-effect as only instruction execution is halted.
> 
> 1.2. For h/w supporting ACPI C2 in which TSC and apic timer don't stop:
> 
> ACPI C2 type is a bit special which is sometimes alias to a deep CPU
> C-state and thus current Xen3.4 treat ACPI C2 type in same manner as
> ACPI C3 type (i.e. broadcast is activated). If user knows on that platform
> ACPI C2 type has not that h/w limitation, 'lapic_timer_c2_ok' could be
> added in grub to deactivate software mitigation.
> 
> 1.3. For the rest implementations support ACPI C2+ in which apic timer
> will be stopped:
> 
> 1.3.1. HPET as broadcast timer source
> 
> HPET can delivery timely wakeup event to CPUs sleep in deep
> C-states with negligible overhead, as stated earlier. But
> HPET mode being used does make some differences to worthy of
> our noting:
> 
> 1.3.1.1. If h/w supports per-channel MSI delivery mode (intr via FSB), it's
> the best broadcast mechanism known so far. No side effect regarding to
> latency, and IPIs used to broadcast wakeup event could be reduced by a factor
> of number of available channels (each channel could independently serve one
> or several sleeping CPUs).
> 
> As long as this feature is available, it's always first prefered automatically
> 
> 1.3.1.2. when MSI delivery mode is absent, we have to use legacy replacement
> mode with only one HPET channel available. Well, it's not that bad as this
> only one channel could serve all sleeping CPUs by using IPIs to wake up.
> However another side-effect occurs, as PIT/RTC interrupts (IRQ0/IRQ8) are
> replaced by HPET channel. Then RTC alarm feature in dom0 will be lost, unless
> we add RTC emulation between dom0's rtc module and Xen's hpet logic (however,
> it's not implemented by far.)
> 
> Due to above side-effect, this broadcast option is disabled by default. In
> that case, PIT broadcast is the default. If user is sure that he doesn't need
> RTC alarm, then use 'hpetbroadcast' grub option to force enabling it.
> 
> 1.3.2. PIT as broadcast timer source
> 
> If MSI based HPET intr delivery is not available or HPET is missing, in all
> cases PIT broadcast is the current default one. As said earlier, 10ms
> periodical mode is implemented on PIT broadcast which thus could incur up to
> 10ms latency for each deep C-state entry/exit. One natural result is to
> observe 'many lost ticks' in some guests.
> 
> 1.4 Suggestions
> 
> So, if user doesn't care about power consumption while his platform does
> expose deep C-states, one mitigation is to add 'max_cstate=' boot option to
> restrict maximum allowed C-states (If limited to C2, ensure adding
> 'lapic_timer_c2_ok' if applied). Runtime modification on 'max_cstate' is
> allowed by xenpm (patch posted in 3/24/2009, not checked in yet).
> 
> If user does care about power consumption w/o requirement on RTC alarm, then
> always using HPET is preferred.
> 
> Last, we could either add RTC emulation on HPET or enhance PIT broadcast to
> use single shot mode, but would like to see comments from community whether
> it's worthy of doing. :-)
> 
> 2. system time/TSC skew
> 
> Similarly to APIC timer stop, TSC is also stopped at deep C-states in some
> implementations, which thus requires Xen to recover lost counts at exit from
> deep C-state by software means. It's easy to think kinds of errors caused by
> software methods. For the detail how TSC skew could occur, its side effects
> and possible solutions, you could refer to our Xen summit presentation:
> http://www.xen.org/files/xensummit_oracle09/XenSummit09pm.pdf
> 
> Below is the brief introduction about which algorithm is available in
> different implementations:
> 
> 2.1. Best case is to have non-stop TSC at h/w implementation level. For
> example, Intel Core-i7 processors supports this green feature which could be
> detected by CPUID. Xen will do nothing once this feature is detected, and thus
> no extra software-caused skew besides dozens of cycles due to crystal drift.
> 
> 2.2. If TSC frequency is invariant across freq/voltage scaling (true for all
> Intel processors supporting VTx), Xen will sync AP's TSCs to BSP's at 1 second
> interval in per-cpu time calibration, meanwhile do recover in a per-cpu style,
> where only elapsed platform counter since last calibration point is
> compensated to local TSC with a boot-time-calculated scale factor. This
> global synchronization along with per-cpu compensation limits TSC skew to ns
> level in most cases.
> 
> 2.3. If TSC frequency is variant across freq/voltage scaling, Xen will only do
> recover in a per-cpu style, where only elapsed platform counter since last
> calibration point is compensated to local TSC with local scale factor. In such
> manner TSC skew across cpus is accumulating and easy to be observed after
> system is up for some time.
> 
> 2.4. Solution
> 
> Once you observe obvious system time/TSC skew, and you don't care power
> consumption specially, then similar to handle broadcast latency:
> 
> Limit 'max_cstate' to C1 or limit 'max_cstate' to a real C2 and give
> 'lapic_timer_c2_ok' option.
> 
> Or, better to run your work on a newer platform with either constant TSC
> frequency or no-stop TSC feature supported. :-)
> 
> Jimmy
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel