Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan <van.freenix@xxxxxxxxx>]

On Thu, Sep 22, 2016 at 10:51:04AM +0100, George Dunlap wrote:
>On 22/09/16 10:27, Peng Fan wrote:
>> On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>>> On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>>>> On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>>>>>
>>>>> Hi Stefano,
>>>>>
>>>>> On 21/09/2016 19:13, Stefano Stabellini wrote:
>>>>>>
>>>>>> On Wed, 21 Sep 2016, Julien Grall wrote:
>>>>>>>
>>>>>>> (CC a couple of ARM folks)
>>>>>>>
>>>>>>> On 21/09/16 11:22, George Dunlap wrote:
>>>>>>>>
>>>>>>>> On 21/09/16 11:09, Julien Grall wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 20/09/16 21:17, Stefano Stabellini wrote:
>>>>>>>>>>
>>>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Stefano,
>>>>>>>>>>>
>>>>>>>>>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>>>>>>>>>>>>>> <van.freenix@xxxxxxxxx>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario
>>>>>>>>>>>>>>> Faggioli
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano
>>>>>>>>>>>>>>>> Stabellini wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>>>>>>>>> I'd like to add a computing capability in
>>>>>>>>>>>>>>> xen/arm, like this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> struct compute_capatiliby
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> ?? char *core_name;
>>>>>>>>>>>>>>> ?? uint32_t rank;
>>>>>>>>>>>>>>> ?? uint32_t cpu_partnum;
>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> struct compute_capatiliby cc=
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> ??{"A72", 4, 0xd08},
>>>>>>>>>>>>>>> ??{"A57", 3, 0xxxx},
>>>>>>>>>>>>>>> ??{"A53", 2, 0xd03},
>>>>>>>>>>>>>>> ??{"A35", 1, ...},
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then when identify cpu, we decide which cpu is
>>>>>>>>>>>>>>> big and which
>>>>>>>>>>>>>>> cpu is
>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>> according to the computing rank.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any comments?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think we definitely need to have Xen have some
>>>>>>>>>>>>>> kind of idea
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> order between processors, so that the user
>>>>>>>>>>>>>> doesn't need to
>>>>>>>>>>>>>> figure out
>>>>>>>>>>>>>> which class / pool is big and which pool is
>>>>>>>>>>>>>> LITTLE.????Whether
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> sort
>>>>>>>>>>>>>> of enumeration is the best way to do that I'll
>>>>>>>>>>>>>> let Julien and
>>>>>>>>>>>>>> Stefano
>>>>>>>>>>>>>> give their opinion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't think an hardcoded list of processor in Xen
>>>>>>>>>>>>> is the right
>>>>>>>>>>>>> solution.
>>>>>>>>>>>>> There are many existing processors and combinations
>>>>>>>>>>>>> for big.LITTLE
>>>>>>>>>>>>> so it
>>>>>>>>>>>>> will
>>>>>>>>>>>>> nearly be impossible to keep updated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would expect the firmware table (device tree,
>>>>>>>>>>>>> ACPI) to provide
>>>>>>>>>>>>> relevant
>>>>>>>>>>>>> data
>>>>>>>>>>>>> for each processor and differentiate big from
>>>>>>>>>>>>> LITTLE core.
>>>>>>>>>>>>> Note that I haven't looked at it for now. A good
>>>>>>>>>>>>> place to start is
>>>>>>>>>>>>> looking
>>>>>>>>>>>>> at
>>>>>>>>>>>>> how Linux does.
>>>>>>>>>>>>
>>>>>>>>>>>> That's right, see
>>>>>>>>>>>> Documentation/devicetree/bindings/arm/cpus.txt. It
>>>>>>>>>>>> is
>>>>>>>>>>>> trivial to identify the two different CPU classes and
>>>>>>>>>>>> which cores
>>>>>>>>>>>> belong
>>>>>>>>>>>> to which class.t, as
>>>>>>>>>>>
>>>>>>>>>>> The class of the CPU can be found from the MIDR, there
>>>>>>>>>>> is no need to
>>>>>>>>>>> use the
>>>>>>>>>>> device tree/acpi for that. Note that I don't think
>>>>>>>>>>> there is an easy
>>>>>>>>>>> way in
>>>>>>>>>>> ACPI (i.e not in AML) to find out the class.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It is harder to figure out which one is supposed to
>>>>>>>>>>>> be
>>>>>>>>>>>> big and which one LITTLE. Regardless, we could
>>>>>>>>>>>> default to using the
>>>>>>>>>>>> first cluster (usually big), which is also the
>>>>>>>>>>>> cluster of the boot
>>>>>>>>>>>> cpu,
>>>>>>>>>>>> and utilize the second cluster only when the user
>>>>>>>>>>>> demands it.
>>>>>>>>>>>
>>>>>>>>>>> Why do you think the boot CPU will usually be a big
>>>>>>>>>>> one? In the case
>>>>>>>>>>> of Juno
>>>>>>>>>>> platform it is configurable, and the boot CPU is a
>>>>>>>>>>> little core on r2
>>>>>>>>>>> by
>>>>>>>>>>> default.
>>>>>>>>>>>
>>>>>>>>>>> In any case, what we care about is differentiate
>>>>>>>>>>> between two set of
>>>>>>>>>>> CPUs. I
>>>>>>>>>>> don't think Xen should care about migrating a guest
>>>>>>>>>>> vCPU between big
>>>>>>>>>>> and
>>>>>>>>>>> LITTLE cpus. So I am not sure why we would want to know
>>>>>>>>>>> that.
>>>>>>>>>>
>>>>>>>>>> No, it is not about migrating (at least yet). It is about
>>>>>>>>>> giving useful
>>>>>>>>>> information to the user. It would be nice if the user had
>>>>>>>>>> to choose
>>>>>>>>>> between "big" and "LITTLE" rather than "class 0x1" and
>>>>>>>>>> "class 0x100", or
>>>>>>>>>> even "A7" or "A15".
>>>>>>>>>
>>>>>>>>> I don't think it is wise to assume that we may have only 2
>>>>>>>>> kind of CPUs
>>>>>>>>> on the platform. We may have more in the future, if so how
>>>>>>>>> would you
>>>>>>>>> name them?
>>>>>>>>
>>>>>>>> I would suggest that internally Xen recognize an arbitrary
>>>>>>>> number of
>>>>>>>> processor "classes", and order them according to more
>>>>>>>> powerful -> less
>>>>>>>> powerful.????Then if at some point someone makes a platform
>>>>>>>> with three
>>>>>>>> processors, you can say "class 0", "class 1" or "class
>>>>>>>> 2".????"big" would
>>>>>>>> be an alias for "class 0" and "little" would be an alias for
>>>>>>>> "class 1".
>>>>>>>
>>>>>>> As mentioned earlier, there is no upstreamed yet device tree
>>>>>>> bindings to know
>>>>>>> the "power" of a CPU (see [1]
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> And in my suggestion, we allow a richer set of labels, so
>>>>>>>> that the user
>>>>>>>> could also be more specific -- e.g., asking for "A15"
>>>>>>>> specifically, for
>>>>>>>> example, and failing to build if there are no A15 cores
>>>>>>>> present, while
>>>>>>>> allowing users to simply write "big" or "little" if they want
>>>>>>>> simplicity
>>>>>>>> / things which work across different platforms.
>>>>>>>
>>>>>>> Well, before trying to do something clever like that (i.e
>>>>>>> naming "big" and
>>>>>>> "little"), we need to have upstreamed bindings available to
>>>>>>> acknowledge the
>>>>>>> difference. AFAICT, it is not yet upstreamed for Device Tree
>>>>>>> (see [1]) and I
>>>>>>> don't know any static ACPI tables providing the similar
>>>>>>> information.
>>>>>>
>>>>>> I like George's idea that "big" and "little" could be just
>>>>>> convenience
>>>>>> aliases. Of course they are predicated on the necessary device
>>>>>> tree
>>>>>> bindings being upstream. We don't need [1] to be upstream in
>>>>>> Linux, just
>>>>>> the binding:
>>>>>>
>>>>>> http://marc.info/?l=linux-arm-kernel&m=147308556729426&w=2
>>>>>>
>>>>>> which has already been acked by the relevant maintainers.
>>>>>
>>>>> This is device tree only. What about ACPI?
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I had few discussions and????more thought about big.LITTLE
>>>>>>> support in Xen. The
>>>>>>> main goal of big.LITTLE is power efficiency by moving task
>>>>>>> around and been
>>>>>>> able to idle one cluster. All the solutions suggested
>>>>>>> (including mine) so far,
>>>>>>> can be replicated by hand (except the VPIDR) so they are mostly
>>>>>>> an automatic
>>>>>>> way. This will also remove the real benefits of big.LITTLE
>>>>>>> because Xen will
>>>>>>> not be able to migrate vCPU across cluster for power
>>>>>>> efficiency.
>>>>>>
>>>>>> The goal of the architects of big.LITTLE might have been power
>>>>>> efficiency, but of course we are free to use any features that
>>>>>> the
>>>>>> hardware provides in the best way for Xen and the Xen community.
>>>>>
>>>>> This is very dependent on how the big.LITTLE has been implemented
>>>>> by the
>>>>> hardware. Some platform can not run both big and LITTLE cores at
>>>>> the same
>>>>> time. You need a proper switch in the firmware/hypervisor.
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> If we care about power efficiency, we would have to handle
>>>>>>> seamlessly
>>>>>>> big.LITTLE in Xen (i.e a guess would only see a kind of CPU).
>>>>>>> This arise quite
>>>>>>> few problem, nothing insurmountable, similar to migration
>>>>>>> across two platforms
>>>>>>> with different micro-architecture (e.g processors): errata,
>>>>>>> features
>>>>>>> supported... The guest would have to know the union of all the
>>>>>>> errata (this is
>>>>>>> done so far via the MIDR, so we would a PV way to do it), and
>>>>>>> only the
>>>>>>> intersection of features would be exposed to the guest. This
>>>>>>> also means the
>>>>>>> scheduler would have to be modified to handle power efficiency
>>>>>>> (not strictly
>>>>>>> necessary at the beginning).
>>>>>>>
>>>>>>> I agree that a such solution would require some work to
>>>>>>> implement, although
>>>>>>> Xen will have a better control of the energy consumption of the
>>>>>>> platform.
>>>>>>>
>>>>>>> So the question here, is what do we want to achieve with
>>>>>>> big.LITTLE?
>>>>>>
>>>>>> I don't think that handling seamlessly big.LITTLE in Xen is the
>>>>>> best way
>>>>>> to do it in the scenarios where Xen on ARM is being used today. I
>>>>>> understand the principles behind it, but I don't think that it
>>>>>> will lead
>>>>>> to good results in a virtualized environment, where there is more
>>>>>> activity and more vcpus than pcpus.
>>>>>
>>>>> Can you detail why you don't think it will give good results?
>>>>>
>>>>>>
>>>>>>
>>>>>> What we discussed in this thread so far is actionable, and gives
>>>>>> us
>>>>>> big.LITTLE support in a short time frame. It is a good fit for
>>>>>> Xen on
>>>>>> ARM use cases and still leads to lower power consumption with an
>>>>>> wise
>>>>>> allocation of big and LITTLE vcpus and pcpus to guests.
>>>>>
>>>>> How this would lead to lower power consumption? If there is nothing
>>>>> running
>>>>> of the processor we would have a wfi loop which will never put the
>>>>> physical
>>>>> CPU in deep sleep. The main advantage of big.LITTLE is too be able
>>>>> to switch
>>>>> off a cluster/cpu when it is not used.
>>>>>
>>>>> Without any knowledge in Xen (such as CPU freq), I am afraid the
>>>>> the power
>>>>> consumption will still be the same.
>>>>>
>>>>>>
>>>>>>
>>>>>> I would start from this approach, then if somebody comes along
>>>>>> with a
>>>>>> plan to implement a big.LITTLE switcher in Xen, I welcome her to
>>>>>> do it
>>>>>> and I would be happy to accept the code in Xen. We'll just make
>>>>>> it
>>>>>> optional.
>>>>>
>>>>> I think we are discussing here a simple design for big.LITTLE. I
>>>>> never asked
>>>>> Peng to do all the work. I am worry that if we start to expose the
>>>>> big.LITTLE
>>>>> to the userspace it will be hard in the future to step back from
>>>>> it.
>>>>
>>>> Hello Julien,
>>>>
>>>>
>>>> I prefer the simple doable method, and As Stefano said, actionable.
>>>>
>>> Yep, this is a very good starting point, IMO.
>>>
>>>> ??- introduce a hypercall interface to let xl can get the different
>>>> classes cpu info.
>>>>
>>> +1
>>>
>>>> ??- Use vcpuclass in xl cfg file to let user create different vcpus
>>>>
>>> Yep.
>>>
>>>> ??- extract cpu computing cap from dts to differentiate cpus. As you
>>>> said, bindings
>>>> ??????not upstreamed. But this is not the hardpoint, we could change the
>>>> info, whether
>>>> ??????dmips or cap in future.
>>>>
>>> Yes (or I should say, "whatever", as I know nothing about all
>>> this! :-P)
>>>
>>>> ??- use cpu hard affinity to block vcpu scheduling bwtween little and
>>>> big pcpu.
>>>>
>>> "to block vcpu scheduling within the specified classes, for each vcpu"
>>>
>>> But, again, yes.
>>>
>>>> ??- block user setting vcpu hard affinity bwtween big and LITTLE.
>>>>
>>> "between the specified class"
>>>
>>> Indeed.
>>>
>>>> ??- only hard affinity seems enough, no need soft affinity.
>>>>
>>> Correct. Just don't care at all and don't touch soft affinity for now.
>>>
>>>> Anyway, for vcpu scheduling bwtween big and LITTLE, if this is the
>>>> right
>>>> direction and you have an idea on how to implement, I could follow
>>>> you on
>>>> enabling this feature with you leading the work. I do not have much
>>>> idea on this.
>>>>
>>> This can come later, either as an enhancement of this affinity based
>>> solution, or being implemented on top of it.
>>>
>>> In any case, let's start with the affinity based solution for now. It's
>>> a good level support already, and a nice first step toward future
>>> improvements.
>>>
>>> Oh, btw, please don't throw away this cpupool patch series either!
>> 
>> Ok -:)
>> 
>>>
>>> A feature like `xl cpupool-biglittle-split' can still be interesting,
>> 
>> "cpupool-cluster-split" maybe a better name?
>
>I think we should name this however we name the different types of cpus.
> i.e., if we're going to call these "cpu classes", then we should call
>this "cpupool-cpuclass-split" or something.

Ok. Got it.

Thanks,
Peng.

>
> -George
>

-- 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel