Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan <van.freenix@xxxxxxxxx>]

On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>> On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>> > 
>> > Hi Stefano,
>> > 
>> > On 21/09/2016 19:13, Stefano Stabellini wrote:
>> > > 
>> > > On Wed, 21 Sep 2016, Julien Grall wrote:
>> > > > 
>> > > > (CC a couple of ARM folks)
>> > > > 
>> > > > On 21/09/16 11:22, George Dunlap wrote:
>> > > > > 
>> > > > > On 21/09/16 11:09, Julien Grall wrote:
>> > > > > > 
>> > > > > > 
>> > > > > > 
>> > > > > > On 20/09/16 21:17, Stefano Stabellini wrote:
>> > > > > > > 
>> > > > > > > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > > > > > > 
>> > > > > > > > Hi Stefano,
>> > > > > > > > 
>> > > > > > > > On 20/09/2016 20:09, Stefano Stabellini wrote:
>> > > > > > > > > 
>> > > > > > > > > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > > > > > > > > 
>> > > > > > > > > > Hi,
>> > > > > > > > > > 
>> > > > > > > > > > On 20/09/2016 12:27, George Dunlap wrote:
>> > > > > > > > > > > 
>> > > > > > > > > > > On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>> > > > > > > > > > > <van.freenix@xxxxxxxxx>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > 
>> > > > > > > > > > > > On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario
>> > > > > > > > > > > > Faggioli
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > 
>> > > > > > > > > > > > > On Mon, 2016-09-19 at 17:01 -0700, Stefano
>> > > > > > > > > > > > > Stabellini wrote:
>> > > > > > > > > > > > > > 
>> > > > > > > > > > > > > > On Tue, 20 Sep 2016, Dario Faggioli wrote:
>> > > > > > > > > > > > I'd like to add a computing capability in
>> > > > > > > > > > > > xen/arm, like this:
>> > > > > > > > > > > > 
>> > > > > > > > > > > > struct compute_capatiliby
>> > > > > > > > > > > > {
>> > > > > > > > > > > > ?? char *core_name;
>> > > > > > > > > > > > ?? uint32_t rank;
>> > > > > > > > > > > > ?? uint32_t cpu_partnum;
>> > > > > > > > > > > > };
>> > > > > > > > > > > > 
>> > > > > > > > > > > > struct compute_capatiliby cc=
>> > > > > > > > > > > > {
>> > > > > > > > > > > > ??{"A72", 4, 0xd08},
>> > > > > > > > > > > > ??{"A57", 3, 0xxxx},
>> > > > > > > > > > > > ??{"A53", 2, 0xd03},
>> > > > > > > > > > > > ??{"A35", 1, ...},
>> > > > > > > > > > > > }
>> > > > > > > > > > > > 
>> > > > > > > > > > > > Then when identify cpu, we decide which cpu is
>> > > > > > > > > > > > big and which
>> > > > > > > > > > > > cpu is
>> > > > > > > > > > > > little
>> > > > > > > > > > > > according to the computing rank.
>> > > > > > > > > > > > 
>> > > > > > > > > > > > Any comments?
>> > > > > > > > > > > 
>> > > > > > > > > > > I think we definitely need to have Xen have some
>> > > > > > > > > > > kind of idea
>> > > > > > > > > > > the
>> > > > > > > > > > > order between processors, so that the user
>> > > > > > > > > > > doesn't need to
>> > > > > > > > > > > figure out
>> > > > > > > > > > > which class / pool is big and which pool is
>> > > > > > > > > > > LITTLE.????Whether
>> > > > > > > > > > > this
>> > > > > > > > > > > sort
>> > > > > > > > > > > of enumeration is the best way to do that I'll
>> > > > > > > > > > > let Julien and
>> > > > > > > > > > > Stefano
>> > > > > > > > > > > give their opinion.
>> > > > > > > > > > 
>> > > > > > > > > > I don't think an hardcoded list of processor in Xen
>> > > > > > > > > > is the right
>> > > > > > > > > > solution.
>> > > > > > > > > > There are many existing processors and combinations
>> > > > > > > > > > for big.LITTLE
>> > > > > > > > > > so it
>> > > > > > > > > > will
>> > > > > > > > > > nearly be impossible to keep updated.
>> > > > > > > > > > 
>> > > > > > > > > > I would expect the firmware table (device tree,
>> > > > > > > > > > ACPI) to provide
>> > > > > > > > > > relevant
>> > > > > > > > > > data
>> > > > > > > > > > for each processor and differentiate big from
>> > > > > > > > > > LITTLE core.
>> > > > > > > > > > Note that I haven't looked at it for now. A good
>> > > > > > > > > > place to start is
>> > > > > > > > > > looking
>> > > > > > > > > > at
>> > > > > > > > > > how Linux does.
>> > > > > > > > > 
>> > > > > > > > > That's right, see
>> > > > > > > > > Documentation/devicetree/bindings/arm/cpus.txt. It
>> > > > > > > > > is
>> > > > > > > > > trivial to identify the two different CPU classes and
>> > > > > > > > > which cores
>> > > > > > > > > belong
>> > > > > > > > > to which class.t, as
>> > > > > > > > 
>> > > > > > > > The class of the CPU can be found from the MIDR, there
>> > > > > > > > is no need to
>> > > > > > > > use the
>> > > > > > > > device tree/acpi for that. Note that I don't think
>> > > > > > > > there is an easy
>> > > > > > > > way in
>> > > > > > > > ACPI (i.e not in AML) to find out the class.
>> > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > It is harder to figure out which one is supposed to
>> > > > > > > > > be
>> > > > > > > > > big and which one LITTLE. Regardless, we could
>> > > > > > > > > default to using the
>> > > > > > > > > first cluster (usually big), which is also the
>> > > > > > > > > cluster of the boot
>> > > > > > > > > cpu,
>> > > > > > > > > and utilize the second cluster only when the user
>> > > > > > > > > demands it.
>> > > > > > > > 
>> > > > > > > > Why do you think the boot CPU will usually be a big
>> > > > > > > > one? In the case
>> > > > > > > > of Juno
>> > > > > > > > platform it is configurable, and the boot CPU is a
>> > > > > > > > little core on r2
>> > > > > > > > by
>> > > > > > > > default.
>> > > > > > > > 
>> > > > > > > > In any case, what we care about is differentiate
>> > > > > > > > between two set of
>> > > > > > > > CPUs. I
>> > > > > > > > don't think Xen should care about migrating a guest
>> > > > > > > > vCPU between big
>> > > > > > > > and
>> > > > > > > > LITTLE cpus. So I am not sure why we would want to know
>> > > > > > > > that.
>> > > > > > > 
>> > > > > > > No, it is not about migrating (at least yet). It is about
>> > > > > > > giving useful
>> > > > > > > information to the user. It would be nice if the user had
>> > > > > > > to choose
>> > > > > > > between "big" and "LITTLE" rather than "class 0x1" and
>> > > > > > > "class 0x100", or
>> > > > > > > even "A7" or "A15".
>> > > > > > 
>> > > > > > I don't think it is wise to assume that we may have only 2
>> > > > > > kind of CPUs
>> > > > > > on the platform. We may have more in the future, if so how
>> > > > > > would you
>> > > > > > name them?
>> > > > > 
>> > > > > I would suggest that internally Xen recognize an arbitrary
>> > > > > number of
>> > > > > processor "classes", and order them according to more
>> > > > > powerful -> less
>> > > > > powerful.????Then if at some point someone makes a platform
>> > > > > with three
>> > > > > processors, you can say "class 0", "class 1" or "class
>> > > > > 2".????"big" would
>> > > > > be an alias for "class 0" and "little" would be an alias for
>> > > > > "class 1".
>> > > > 
>> > > > As mentioned earlier, there is no upstreamed yet device tree
>> > > > bindings to know
>> > > > the "power" of a CPU (see [1]
>> > > > 
>> > > > > 
>> > > > > 
>> > > > > And in my suggestion, we allow a richer set of labels, so
>> > > > > that the user
>> > > > > could also be more specific -- e.g., asking for "A15"
>> > > > > specifically, for
>> > > > > example, and failing to build if there are no A15 cores
>> > > > > present, while
>> > > > > allowing users to simply write "big" or "little" if they want
>> > > > > simplicity
>> > > > > / things which work across different platforms.
>> > > > 
>> > > > Well, before trying to do something clever like that (i.e
>> > > > naming "big" and
>> > > > "little"), we need to have upstreamed bindings available to
>> > > > acknowledge the
>> > > > difference. AFAICT, it is not yet upstreamed for Device Tree
>> > > > (see [1]) and I
>> > > > don't know any static ACPI tables providing the similar
>> > > > information.
>> > > 
>> > > I like George's idea that "big" and "little" could be just
>> > > convenience
>> > > aliases. Of course they are predicated on the necessary device
>> > > tree
>> > > bindings being upstream. We don't need [1] to be upstream in
>> > > Linux, just
>> > > the binding:
>> > > 
>> > > http://marc.info/?l=linux-arm-kernel&m=147308556729426&w=2
>> > > 
>> > > which has already been acked by the relevant maintainers.
>> > 
>> > This is device tree only. What about ACPI?
>> > 
>> > > 
>> > > 
>> > > 
>> > > > 
>> > > > I had few discussions and????more thought about big.LITTLE
>> > > > support in Xen. The
>> > > > main goal of big.LITTLE is power efficiency by moving task
>> > > > around and been
>> > > > able to idle one cluster. All the solutions suggested
>> > > > (including mine) so far,
>> > > > can be replicated by hand (except the VPIDR) so they are mostly
>> > > > an automatic
>> > > > way. This will also remove the real benefits of big.LITTLE
>> > > > because Xen will
>> > > > not be able to migrate vCPU across cluster for power
>> > > > efficiency.
>> > > 
>> > > The goal of the architects of big.LITTLE might have been power
>> > > efficiency, but of course we are free to use any features that
>> > > the
>> > > hardware provides in the best way for Xen and the Xen community.
>> > 
>> > This is very dependent on how the big.LITTLE has been implemented
>> > by the
>> > hardware. Some platform can not run both big and LITTLE cores at
>> > the same
>> > time. You need a proper switch in the firmware/hypervisor.
>> > 
>> > > 
>> > > 
>> > > > 
>> > > > If we care about power efficiency, we would have to handle
>> > > > seamlessly
>> > > > big.LITTLE in Xen (i.e a guess would only see a kind of CPU).
>> > > > This arise quite
>> > > > few problem, nothing insurmountable, similar to migration
>> > > > across two platforms
>> > > > with different micro-architecture (e.g processors): errata,
>> > > > features
>> > > > supported... The guest would have to know the union of all the
>> > > > errata (this is
>> > > > done so far via the MIDR, so we would a PV way to do it), and
>> > > > only the
>> > > > intersection of features would be exposed to the guest. This
>> > > > also means the
>> > > > scheduler would have to be modified to handle power efficiency
>> > > > (not strictly
>> > > > necessary at the beginning).
>> > > > 
>> > > > I agree that a such solution would require some work to
>> > > > implement, although
>> > > > Xen will have a better control of the energy consumption of the
>> > > > platform.
>> > > > 
>> > > > So the question here, is what do we want to achieve with
>> > > > big.LITTLE?
>> > > 
>> > > I don't think that handling seamlessly big.LITTLE in Xen is the
>> > > best way
>> > > to do it in the scenarios where Xen on ARM is being used today. I
>> > > understand the principles behind it, but I don't think that it
>> > > will lead
>> > > to good results in a virtualized environment, where there is more
>> > > activity and more vcpus than pcpus.
>> > 
>> > Can you detail why you don't think it will give good results?
>> > 
>> > > 
>> > > 
>> > > What we discussed in this thread so far is actionable, and gives
>> > > us
>> > > big.LITTLE support in a short time frame. It is a good fit for
>> > > Xen on
>> > > ARM use cases and still leads to lower power consumption with an
>> > > wise
>> > > allocation of big and LITTLE vcpus and pcpus to guests.
>> > 
>> > How this would lead to lower power consumption? If there is nothing
>> > running
>> > of the processor we would have a wfi loop which will never put the
>> > physical
>> > CPU in deep sleep. The main advantage of big.LITTLE is too be able
>> > to switch
>> > off a cluster/cpu when it is not used.
>> > 
>> > Without any knowledge in Xen (such as CPU freq), I am afraid the
>> > the power
>> > consumption will still be the same.
>> > 
>> > > 
>> > > 
>> > > I would start from this approach, then if somebody comes along
>> > > with a
>> > > plan to implement a big.LITTLE switcher in Xen, I welcome her to
>> > > do it
>> > > and I would be happy to accept the code in Xen. We'll just make
>> > > it
>> > > optional.
>> > 
>> > I think we are discussing here a simple design for big.LITTLE. I
>> > never asked
>> > Peng to do all the work. I am worry that if we start to expose the
>> > big.LITTLE
>> > to the userspace it will be hard in the future to step back from
>> > it.
>> 
>> Hello Julien,
>> 
>> 
>> I prefer the simple doable method, and As Stefano said, actionable.
>> 
>Yep, this is a very good starting point, IMO.
>
>> ??- introduce a hypercall interface to let xl can get the different
>> classes cpu info.
>>
>+1
>
>> ??- Use vcpuclass in xl cfg file to let user create different vcpus
>>
>Yep.
>
>> ??- extract cpu computing cap from dts to differentiate cpus. As you
>> said, bindings
>> ??????not upstreamed. But this is not the hardpoint, we could change the
>> info, whether
>> ??????dmips or cap in future.
>>
>Yes (or I should say, "whatever", as I know nothing about all
>this! :-P)
>
>> ??- use cpu hard affinity to block vcpu scheduling bwtween little and
>> big pcpu.
>>
>"to block vcpu scheduling within the specified classes, for each vcpu"
>
>But, again, yes.
>
>> ??- block user setting vcpu hard affinity bwtween big and LITTLE.
>>
>"between the specified class"
>
>Indeed.
>
>> ??- only hard affinity seems enough, no need soft affinity.
>> 
>Correct. Just don't care at all and don't touch soft affinity for now.
>
>> Anyway, for vcpu scheduling bwtween big and LITTLE, if this is the
>> right
>> direction and you have an idea on how to implement, I could follow
>> you on
>> enabling this feature with you leading the work. I do not have much
>> idea on this.
>> 
>This can come later, either as an enhancement of this affinity based
>solution, or being implemented on top of it.
>
>In any case, let's start with the affinity based solution for now. It's
>a good level support already, and a nice first step toward future
>improvements.
>
>Oh, btw, please don't throw away this cpupool patch series either!

Ok -:)

>
>A feature like `xl cpupool-biglittle-split' can still be interesting,

"cpupool-cluster-split" maybe a better name?

>completely orthogonally and independently from the affinity based work,
>and this series looks like it can be used to implement that. :-)

Agree. All pcpus default can be assigned into cpupool0 based on the affinity 
work.
We could add one like "cpupool-numa-split" to split different classes cpu
into different cpupools.

Thanks,
Peng.

>
>Thanks and Regards,
>Dario
>-- 
><<This happens because I choose it to happen!>> (Raistlin Majere)
>-----------------------------------------------------------------
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)



-- 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel