Re: [RFC PATCH 0/2] Boot time cpupools

[Julien Grall <julien@xxxxxxx>]

(Prunning some CC to just leave Arm and sched folks)

On 17/11/2021 12:07, Bertrand Marquis wrote:
> Hi Julien,
Hi Bertrand,

> > On 17 Nov 2021, at 11:48, Julien Grall <julien@xxxxxxx> wrote:
> >
> > On 17/11/2021 11:16, Bertrand Marquis wrote:
> > > Hi Julien,
> > Hi,
> >
> > > > On 17 Nov 2021, at 10:26, Julien Grall <julien@xxxxxxx> wrote:
> > > >
> > > > Hi Luca,
> > > >
> > > > On 17/11/2021 09:57, Luca Fancellu wrote:
> > > > > Currently Xen creates a default cpupool0 that contains all the cpu brought up
> > > > > during boot and it assumes that the platform has only one kind of CPU.
> > > > > This assumption does not hold on big.LITTLE platform, but putting different
> > > > > type of CPU in the same cpupool can result in instability and security issues
> > > > > for the domains running on the pool.
> > > > I agree that you can't move a LITTLE vCPU to a big pCPU. However...
> > > >
> > > > > For this reason this serie introduces an architecture specific way to create
> > > > > different cpupool at boot time, this is particularly useful on ARM big.LITTLE
> > > > > platform where there might be the need to have different cpupools for each type
> > > > > of core, but also systems using NUMA can have different cpu pool for each node.
> > > > ... from my understanding, all the vCPUs of a domain have to be in the same 
> > > > cpupool. So with this approach it is not possible:
> > > >    1) to have a mix of LITTLE and big vCPUs in the domain
> > > >    2) to create a domain spanning across two NUMA nodes
> > > >
> > > > So I think we need to make sure that any solutions we go through will not 
> > > > prevent us to implement those setups.
> > > The point of this patch is to make all cores available without breaking the 
> > > current behaviour of existing system.
> > I might be missing some context here. By breaking current behavior, do you mean user that 
> > may want to add "hmp-unsafe" on the command line?
> Right, with hmp-unsafe the behaviour is now the same as without, only extra 
> cores are parked in other cpupools.
>
> So you have a point in fact that behaviour is changed for someone who was using 
> hmp-unsafe before if this is activated.
> The command line argument suggested by Jurgen definitely makes sense here.
>
> We could instead do the following:
> - when this is activated in the configuration, boot all cores and park them in 
> different pools (depending on command line argument). Current behaviour not 
> change if other pools are not used (but more cores will be on in xen)
From my understanding, it is possible to move a pCPU in/out a pool 
afterwards. So the security concern with big.LITTLE is still present, 
even though it would be difficult to hit it.
I am also concerned that it would be more difficult to detect any 
misconfiguration. So I think this option would still need to be turned 
on only if hmp-unsafe are the new command line one are both on.
If we want to enable it without hmp-unsafe on, we would need to at least 
lock the pools.
> - when hmp-unsafe is on, this feature will be turned of (if activated in 
> configuration) and all cores would be added in the same pool.
>
> What do you think ?
I am split. On one hand, this is making easier for someone to try 
big.LITTLE as you don't have manually pin vCPUs. On the other hand, this 
is handling a single use-case and you would need to use hmp-unsafe and 
pinning if you want to get more exotic setup (e.g. a domain with 
big.LITTLE).
Another possible solution is to pin dom0 vCPUs (AFAIK they are just 
sticky by default) and then create the pools from dom0 userspace. My 
assumption is for dom0less we would want to use pinning instead.
That said I would like to hear from Xilinx and EPAM as, IIRC, they are 
already using hmp-unsafe in production.
Cheers,

--
Julien Grall