Re: [Xen-devel] Workings/effectiveness of the xen-acpi-processor driver

[Stefan Bader <stefan.bader@xxxxxxxxxxxxx>]

Sorry for not responding sooner, there was a good opportunity for a long week
end here I could not miss. Also I had been moving the power meter over to the
other machine.

On 01.05.2012 22:02, Konrad Rzeszutek Wilk wrote:
> On Thu, Apr 26, 2012 at 06:25:28PM +0200, Stefan Bader wrote:
>> On 26.04.2012 17:50, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Apr 25, 2012 at 03:00:58PM +0200, Stefan Bader wrote:
>>>> Since there have been requests about that driver to get backported into 
>>>> 3.2, I
>>>> was interested to find out what or how much would be gained by that.
>>>>
>>>> The first system I tried was an AMD based one (8 core Opteron 6128@2GHz). 
>>>> Which
>>>> was not very successful as the drivers bail out of the init function 
>>>> because the
>>>> first call to acpi_processor_register_performance() returns -ENODEV. There 
>>>> is
>>>> some frequency scaling when running without Xen, so I need to do some more
>>>> debugging there.
>>>
>>> Did you back-port the other components - the ones that turn off the native
>>> frequency scalling?
>>>
>>>       provide disable_cpufreq() function to disable the API.
>>>     xen/acpi-processor: Do not depend on CPU frequency scaling drivers.
>>>       xen/cpufreq: Disable the cpu frequency scaling drivers from loading
>>>>
>>
>> Yes, here is the full set for reference:
>>
>> * xen/cpufreq: Disable the cpu frequency scaling drivers from loading.
>> * xen/acpi: Remove the WARN's as they just create noise.
>> * xen/acpi: Fix Kconfig dependency on CPU_FREQ
>> * xen/acpi-processor: Do not depend on CPU frequency scaling drivers.
>> * xen/acpi-processor: C and P-state driver that uploads said data to hyper
>> * provide disable_cpufreq() function to disable the API.
> 
> And (Linus just pulled it), you also need this one:
>  df88b2d96e36d9a9e325bfcd12eb45671cbbc937 (xen/enlighten: Disable MWAIT_LEAF 
> so that acpi-pad won't be loaded.)

I suspect that one only if I would have

commit 73c154c60be106b47f15d1111fc2d75cc7a436f2
xen/enlighten: Expose MWAIT and MWAIT_LEAF if hypervisor OKs it.

and the change in the hypervisor to implement XEN_SET_PDC.

> 
>>
>>>> The second system was an Intel one (4 core i7 920@xxxxxxx) which was
>>>> successfully loading the driver. Via xenpm I can see the various 
>>>> frequencies and
>>>> also see them being changed. However the cpuidle data out of xenpm looks a 
>>>> bit odd:
>>>>
>>>> #> xenpm get-cpuidle-states 0
>>>> Max C-state: C7
>>>>
>>>> cpu id               : 0
>>>> total C-states       : 2
>>>> idle time(ms)        : 10819311
>>>> C0                   : transition [00000000000000000001]
>>>>                        residency  [00000000000000005398 ms]
>>>> C1                   : transition [00000000000000000001]
>>>>                        residency  [00000000000010819311 ms]
>>>> pc3                  : [00000000000000000000 ms]
>>>> pc6                  : [00000000000000000000 ms]
>>>> pc7                  : [00000000000000000000 ms]
>>>> cc3                  : [00000000000000000000 ms]
>>>> cc6                  : [00000000000000000000 ms]
>>>>
>>>> Also gathering samples over 30s does look like only C0 and C1 are used. 
>>>> This
>>>
>>> Yes. 
>>>> might be because C1E support is enabled in BIOS but when looking at the
>>>> intel_idle data in sysfs when running without a hypervisor will show C3 
>>>> and C6
>>>> for the cores. That could have been just a wrong output, so I plugged in a 
>>>> power
>>>> meter and compared a kernel running natively and running as dom0 (with and
>>>> without the acpi-processor driver).
>>>>
>>>> Native: 175W
>>>> dom0:   183W (with only marginal difference between with or without the
>>>>               processor driver)
>>>> [yes, the system has a somewhat high base consumption which I attribute to 
>>>> a
>>>> ridiculously dimensioned graphics subsystem to be running a text console]
>>>>
>>>> This I would take as C3 and C6 really not being used and the frequency 
>>>> scaling
> 
> So the other thing I forgot to note is that C3->C6 have a detrimental
> effect on some Intel boxes with Xen. We haven't figured out exactly which ones
> and the bug is definitly in the hypervisor. The bug is that when the CPU goes 
> in
> those states the NIC ends up being unresponsive. Its like the interrupts 
> stopped
> being ACKed. If I run 'xenpm set-max-cstate 2' the issue disappears.
> 

I think I saw weird behavior not only under Xen with that. One netbook I got has
the nasty tendency to go into undetermined sleeps and you had to hit some keys
to keep it running. In that case I found out that the intel gfx card was one
which does not cause interrupts on vsync and other sources seemed not able to
wake up from deeper c-states when being msi. So the wonder cure there is to
force interrupts not being msi.

That aside, I was also comparing power usage in idle on the AMD box. And that
shows an even bigger difference between running natively and running dom0 under
the hypervisor (103W against 127W). As that is AMD there never are other
c-states than c1(e) which xenpm shows as being used (got the feeling that with
AMD CPUs it is quite hard to find out the c1 residency nowadays (only intel_idle
keeps statistics). I think I got some data now but still need to parse it (if I
am not wrong one needs to use power trace now).
But regardless it would hint that having the deeper c-states may not be the real
solution.

>>>
>>> To go in deeper modes there is also a need to backport a Xen unstable
>>> hypercall which will allow the kernel to detect the other states besides
>>> C0-C2.
>>>
>>> "XEN_SET_PDC query was implemented in c/s 23783:
>>>     "ACPI: add _PDC input override mechanism".
>>>     
>>
>> I see. There is a kernel patch about enabling MWAIT that refers to that...
> 
> Were there any special things you ran when checking the output? Just plugging
> and looking at the results?

In the first step it only was the idle power usage. But as you say the real
target for the patches were a better overall performance under use. So I need to
think of something to quantify the difference.

>>
>>>
>>>> having no impact on the idle system is not that much surprising. But if 
>>>> that was
>>>> true it would also limit the usefulness of the turbo mode which I 
>>>> understand
>>>> would also be limited by the c-state of the other cores.
>>>
>>> Hm, I should double-check that - but somehow I thought that Xen independetly
>>> checks for TurboMode and if the P-states are in, then they are activated.
> 
> I did a bit of checking around and it does seem that is the case. From what
> I have gathered the TurboMode kicks in when the CPU is C0 mode (which should
> be obvious), and when the other cores are in anything but C0 mode. And sure
> enough that seems to be the case. But I can't get the concrete details whether
> the "but C0 mode" means that TurboMode will work better if the C mode is 
> legacy
> C1, C2, C3 or the CPU C-states (so MWAIT enabled). Trying to find out from
> Len Brown more details..
>>>

Right, that is how I understood things, too. Depending on how many cores are
"active" (which does not really say what active means) the frequency can go up
in steps of 133Mhz (or was that 200) up to a certain upper limit. The upper
limit depends on how many cores are active but for my CPU seems just one more
step between 2-4 cores active and 1 core active.
The rest is not under OS control beside the CPU to go up in frequency need to be
in P0 mode. From the looks other cores can be in P0 as well and can go up a bit
as well.

>> Turbo mode should be enabled. I had been only looking at a generic overview
>> about it on Intel site which sounded like it  would make more of a 
>> difference on
>> how much one core could get overclocked related to how many cores are active
>> (and I translated active or not into deeper c-states or not).
>> Looking at the verbose output of turbostat it seems not to make that much
>> difference whether 2-4 cores are running. A single core alone could get one 
>> more
>> increment in clock stepping. That does not immediately sound a lot. And of
>> course how much or long the higher clock is used depends on other factors as
>> well and is not under OS control.
>>
>> In the end it is probably quite dynamic and hard to come up with hard facts 
>> to
>> prove its value. Though if I can lower the idle power usage by reaching a bit
>> further, that would greatly help to justify the effort and potential risk of
>> backporting...
> 
> I understand. I wish I could give you the exact percentage points by which
> the power usage will drop. But I think the more substantial reason benefit of
> these patches is performance gains. The ones that Ian Campbell ran and were
> posted on Phorenix site paint that they are beneficial.

I probably should ask Ian (or check on that post) what tests he was using to
measure the difference.

> 
>>
>>>>
>>>> Do I misread the data I see? Or maybe its a known limitation? In case it is
>>>> worth doing more research I'll gladly try things and gather more data.
>>>
>>> Just missing some patches. 
>>>
>>> Oh, and this one:
>>>       xen/acpi: Fix Kconfig dependency on CPU_FREQ
>>>
>>> Hmm.. I think a patch disappeared somewhere.
> 
> That was the one I referenced at the beginning of this email.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel