Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 19 Sep 2016, Peng Fan wrote:
> On Mon, Sep 19, 2016 at 11:59:05AM +0200, Julien Grall wrote:
> >
> >
> >On 19/09/2016 11:38, Peng Fan wrote:
> >>On Mon, Sep 19, 2016 at 10:53:56AM +0200, Julien Grall wrote:
> >>>Hello,
> >>>
> >>>On 19/09/2016 10:36, Peng Fan wrote:
> >>>>On Mon, Sep 19, 2016 at 10:09:06AM +0200, Julien Grall wrote:
> >>>>>Hello Peng,
> >>>>>
> >>>>>On 19/09/2016 04:08, van.freenix@xxxxxxxxx wrote:
> >>>>>>From: Peng Fan <peng.fan@xxxxxxx>
> >>>>>>
> >>>>>>This patchset is to support XEN run on big.little SoC.
> >>>>>>The idea of the patch is from
> >>>>>>"https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg00465.html";
> >>>>>>
> >>>>>>There are some changes to cpupool and add x86 stub functions to avoid 
> >>>>>>build
> >>>>>>break. Sending The RFC patchset out is to request for comments to see 
> >>>>>>whether
> >>>>>>this implementation is acceptable or not. Patchset have been tested 
> >>>>>>based on
> >>>>>>xen-4.8 unstable on NXP i.MX8.
> >>>>>>
> >>>>>>I use Big/Little CPU and cpupool to explain the idea.
> >>>>>>A pool contains Big CPUs is called Big Pool.
> >>>>>>A pool contains Little CPUs is called Little Pool.
> >>>>>>If a pool does not contains any physical cpus, Little CPUs or Big CPUs
> >>>>>>can be added to the cpupool. But the cpupool can not contain both Little
> >>>>>>and Big CPUs. The CPUs in a cpupool must have the same cpu type(midr 
> >>>>>>value for ARM).
> >>>>>>CPUs can not be added to the cpupool which contains cpus that have 
> >>>>>>different cpu type.
> >>>>>>Little CPUs can not be moved to Big Pool if there are Big CPUs in Big 
> >>>>>>Pool,
> >>>>>>and versa. Domain in Big Pool can not be migrated to Little Pool, and 
> >>>>>>versa.
> >>>>>>When XEN tries to bringup all the CPUs, only add CPUs with the same cpu 
> >>>>>>type(same midr value)
> >>>>>>into cpupool0.
> >>>>>
> >>>>>As mentioned in the mail you pointed above, this series is not enough to 
> >>>>>make
> >>>>>big.LITTLE working on then. Xen is always using the boot CPU to detect 
> >>>>>the
> >>>>>list of features. With big.LITTLE features may not be the same.
> >>>>>
> >>>>>And I would prefer to see Xen supporting big.LITTLE correctly before
> >>>>>beginning to think to expose big.LITTLE to the userspace (via cpupool)
> >>>>>automatically.
> >>>>
> >>>>Do you mean vcpus be scheduled between big and little cpus freely?
> >>>
> >>>By supporting big.LITTLE correctly I meant Xen thinks that all the cores 
> >>>has
> >>>the same set of features. So the feature detection is only done the boot 
> >>>CPU.
> >>>See processor_setup for instance...
> >>>
> >>>Moving vCPUs between big and little cores would be a hard task (cache line
> >>>issue, and possibly feature) and I don't expect to ever cross this in Xen.
> >>>However, I am expecting to see big.LITTLE exposed to the guest (i.e having
> >>>big and little vCPUs).
> >>
> >>big vCPUs scheduled on big Physical CPUs and little vCPUs scheduled on 
> >>little
> >>physical cpus, right?
> >>If it is, is there is a need to let Xen think all the cores has the same set
> >>of features?
> >
> >I think you missed my point. The feature registers on big and little cores
> >may be different. Currently, Xen is reading the feature registers of the CPU
> >boot and wrongly assumes that those features will exists on all CPUs. This is
> >not the case and should be fixed before we are getting in trouble.
> >
> >>
> >>Developing big.little guest support, I am not sure how much efforts needed.
> >>Is this really needed?
> >
> >This is not necessary at the moment, although I have seen some interest about
> >it. Running a guest only on a little core is a nice beginning, but a guest
> >may want to take advantage of big.LITTLE (running hungry app on big one and
> >little on small one).
> >
> >>
> >>>
> >>>>
> >>>>This patchset is to use cpupool to block the vcpu be scheduled between 
> >>>>big and
> >>>>little cpus.
> >>>>
> >>>>>
> >>>>>See for instance v->arch.actlr = READ_SYSREG32(ACTLR_EL1).
> >>>>
> >>>>Thanks for this. I only expose cpuid to guest, missed actlr. I'll check
> >>>>the A53 and A72 TRM about AArch64 implementationd defined registers.
> >>>>This actlr can be added to the cpupool_arch_info as midr.
> >>>>
> >>>>Reading "vcpu_initialise", seems only MIDR and ACTLR needs to be handled.
> >>>>Please advise if I missed anything else.
> >>>
> >>>Have you check the register emulation?
> >>
> >>Checked midr. Have not checked others.
> >>I think I missed some registers in ctxt_switch_to.
> >>
> >>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>>Thinking an SoC with 4 A53(cpu[0-3]) + 2 A72(cpu[4-5]), cpu0 is the 
> >>>>>>first one
> >>>>>>that boots up. When XEN tries to bringup secondary CPUs, add cpu[0-3] to
> >>>>>>cpupool0 and leave cpu[4-5] not in any cpupool. Then when Dom0 boots up,
> >>>>>>`xl cpupool-list -c` will show cpu[0-3] in Pool-0.
> >>>>>>
> >>>>>>Then use the following script to create a new cpupool and add cpu[4-5] 
> >>>>>>to
> >>>>>>the cpupool.
> >>>>>>#xl cpupool-create name=\"Pool-A72\" sched=\"credit2\"
> >>>>>>#xl cpupool-cpu-add Pool-A72 4
> >>>>>>#xl cpupool-cpu-add Pool-A72 5
> >>>>>>#xl create -d /root/xen/domu-test pool=\"Pool-A72\"
> >>>>>
> >>>>>I am a bit confused with these runes. It means that only the first kind 
> >>>>>of
> >>>>>CPUs have pool assigned. Why don't you directly create all the pools at 
> >>>>>boot
> >>>>>time?
> >>>>
> >>>>If need to create all the pools, need to decided how many pools need to 
> >>>>be created.
> >>>>I thought about this, but I do not come out a good idea.
> >>>>
> >>>>The cpupool0 is defined in xen/common/cpupool.c, if need to create many 
> >>>>pools,
> >>>>need to alloc cpupools dynamically when booting. I would not like to 
> >>>>change a
> >>>>lot to common code.
> >>>
> >>>Why? We should avoid to choose a specific design just because the common 
> >>>code
> >>>does not allow you to do it without heavy change.
> >>>
> >>>We never came across the big.LITTLE problem on x86, so it is normal to 
> >>>modify
> >>>the code.
> >>>
> >>>>The implementation in this patchset I think is an easy way to let Big and 
> >>>>Little
> >>>>CPUs all run.
> >>>
> >>>I care about having a design allowing an easy use of big.LITTLE on Xen. 
> >>>Your
> >>>solution requires the administrator to know the underlying platform and
> >>>create the pool.
> >>
> >>I suppose big.little is mainly used in embedded SoC :). So the 
> >>user(developer?)
> >>needs to know the hardware platform.
> >
> >The user will always be happy if Xen can save him a bit of time to create
> >cpupool. ;)
> >
> >>
> >>>
> >>>In the solution I suggested, the pools would be created by Xen (and the 
> >>>info
> >>>exposed to the userspace for the admin).
> >>
> >>I think the reason to create cpupools to support big.little SoC is to
> >>avoid vcpus scheduled between big and little physical cpus.
> >>
> >>If need to support big.little guest, I think no need to create more
> >>cpupools expect cpupoo0. Need to make sure vcpus not be scheduled between
> >>big and little physical cpus. All the cpus needs to be in one cpupool.
> >>
> >>>
> >>>>
> >>>>>
> >>>>>Also, in which pool a domain will be created if none is specified?
> >>>>>
> >>>>>>Now `xl cpupool-list -c` shows:
> >>>>>>Name            CPU list
> >>>>>>Pool-0          0,1,2,3
> >>>>>>Pool-A72        4,5
> >>>>>>
> >>>>>>`xl cpupool-list` shows:
> >>>>>>Name               CPUs   Sched     Active   Domain count
> >>>>>>Pool-0               4    credit       y          1
> >>>>>>Pool-A72             2   credit2       y          1
> >>>>>>
> >>>>>>`xl cpupool-cpu-remove Pool-A72 4`, then `xl cpupool-cpu-add Pool-0 4`
> >>>>>>not success, because Pool-0 contains A53 CPUs, but CPU4 is an A72 CPU.
> >>>>>>
> >>>>>>`xl cpupool-migrate DomU Pool-0` will also fail, because DomU is created
> >>>>>>in Pool-A72 with A72 vcpu, while Pool-0 have A53 physical cpus.
> >>>>>>
> >>>>>>Patch 1/5:
> >>>>>>use "cpumask_weight(cpupool0->cpu_valid);" to replace 
> >>>>>>"num_online_cpus()",
> >>>>>>because num_online_cpus() counts all the online CPUs, but now we only
> >>>>>>need Big or Little CPUs.
> >>>>>
> >>>>>So if I understand correctly, if the boot CPU is a little CPU, DOM0 will
> >>>>>always be able to only use little ones. Is that right?
> >>>>
> >>>>Yeah. Dom0 only use the little ones.
> >>>
> >>>This is really bad, dom0 on normal case will have all the backends. It may
> >>>not be possible to select the boot CPU, and therefore always get a little
> >>>CPU.
> >>
> >>Dom0 runs in cpupool0. cpupool0 only contains the cpu[0-3] in my case.
> >
> >So the performance of dom0 will be impacted because it will only use little
> >cores.
> >
> >>
> >>>
> >>>Creating the pool at boot time would have avoid a such issue because, 
> >>>unless
> >>>we expose big.LITTLE to dom0 (I would need the input of George and Dario 
> >>>for
> >>>this bits), we could have a parameter to specify which set of CPUs (e.g 
> >>>pool)
> >>>to allocate dom0 vCPUs.
> >>
> >>dom0 is control domain, I think no need to expose big.little for dom0.
> >>Pin VCPU to specific physical cpus, this may help support big.little guest.
> >>
> >>>
> >>>Note, that I am not asking you to implement everything. But I think we 
> >>>need a
> >>>coherent view of big.LITTLE support in Xen today to go forward.
> >>
> >>Yeah. Then you prefer supporting big.little guest?
> >
> >I have seen some interest on it.
> 
> I have no idea about the use cases. Anyway big.little guest do have benefits.
> 
> >
> >>Please advise if you have any plan/ideas or what I can do on this.
> >
> >I already gave some ideas on what could be done for big.LITTLE support. But,
> >I admit I haven't yet much think about it, so I may miss some part.
> 
> This comes to me a question about Dom0. Do we need to let Dom0 be big.little? 
> or
> add a bootargs to indicate big.little, big or little for Dom0?

These are my two cents: the two cpupools (one big, one LITTLE) need to
be created automatically, either by Xen or by an xl command, such as "xl
cpupool-bigLITTLE-split", as George suggested. If done with an xl
command, it needs to be called automatically at boot somehow, otherwise
we'll leave half of the cpus unused by default. As Xen needs to be able
to recognize big.LITTLE and only start Dom0 on big (or LITTLE) cpus, I
think it might make sense to create both cpupools directly in Xen.

It should be possible to configure to which cpupool Dom0 belongs to, but
I would default to big for performance reasons.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel