Re: [Xen-devel] [PATCH v4 21/21] xl: vNUMA support

[Wei Liu <wei.liu2@xxxxxxxxxx>]

On Wed, Jan 28, 2015 at 04:46:05PM +0000, Ian Campbell wrote:
> On Fri, 2015-01-23 at 11:13 +0000, Wei Liu wrote:
> > This patch includes configuration options parser and documentation.
> > 
> > Please find the hunk to xl.cfg.pod.5 for more information.
> > 
> > Signed-off-by: Wei Liu <wei.liu2@xxxxxxxxxx>
> > Cc: Ian Campbell <ian.campbell@xxxxxxxxxx>
> > Cc: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
> > Cc: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
> > Cc: Elena Ufimtseva <ufimtseva@xxxxxxxxx>
> > ---
> > Changes in v2:
> > 1. Make vnuma_vdistances mandatory.
> > 2. Use nested list to specify vdistances.
> > 3. Update documentation.
> > ---
> >  docs/man/xl.cfg.pod.5    |   39 ++++++++++++
> >  tools/libxl/xl_cmdimpl.c |  147 
> > ++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 186 insertions(+)
> > 
> > diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> > index e2f91fc..b23bd6f 100644
> > --- a/docs/man/xl.cfg.pod.5
> > +++ b/docs/man/xl.cfg.pod.5
> > @@ -266,6 +266,45 @@ it will crash.
> >  
> >  =back
> >  
> > +=head3 Virtual NUMA Memory Allocation
> > +
> > +=over 4
> > +
> > +=item B<vnuma_memory=[ NUMBER, NUMBER, ... ]>
> > +
> > +Specify the size of memory covered by each virtual NUMA node. The number of
> > +elements in the list also implicitly defines the number of virtual NUMA 
> > nodes.
> > +
> > +The sum of all elements in this list should be equal to memory size 
> > specified
> > +by B<maxmem=> in guest configuration file, or B<memory=> if B<maxmem=> is 
> > not
> > +specified.
> 
> Could we make it permissible to omit memory= and have the toolstack do
> the maths for us?
> 

This is OK.

> I suppose == maxmem is due to no vnuma vs. PoD? What about for PV

Yes.

> guests, is preballooning allowed there too?

I need to check PV boot sequence to have a definite answer.

Currently memory allocation in libxc only deals with a chunk of
contiguous memory. Not sure if I change that will break some assumptions
that guest kernel makes.

> 
> > +
> > +=item B<vnuma_vcpu_map=[ NUMBER, NUMBER, ... ]>
> > +
> > +Specifiy which virutal NUMA node a specific vcpu belongs to. The number of
> 
> "Specify" and "virtual".
> 
> > +elements in this list should be equal to B<maxvcpus=> in guest 
> > configuration
> > +file, or B<vcpus=> if B<maxvcpus=> is not specified.
> 
> Again, can we relieve the user of these tedious sums?

This point becomes moot if we use other syntax to specify vcpu list.

> > +=item B<vnuma_pnode_map=[ NUMBER, NUMBER, ... ]>
> > +
> > +Specifiy which physical NUMA node a specific virtual NUMA node maps to. The
> 
> "Specify" again.
> 
> > +number of elements in this list should be equal to the number of virtual
> > +NUMA nodes defined in B<vnuma_memory=>.
> 
> Would it make sense to instead have a single array or e.g. "NODE:SIZE"
> or something?
> 

Or "PNODE:SIZE:VCPUS"?

> > +
> > +=item B<vnuma_vdistance=[ [NUMBER, ..., NUMBER], [NUMBER, ..., NUMBER], 
> > ... ]>
> > +
> > +Two dimensional list to specify distances among nodes.
> > +
> > +The number of elements in the first dimension list equals the number of 
> > virtual
> > +nodes. Each element in position B<i> is a list that specifies the distances
> > +from node B<i> to other nodes.
> > +
> > +For example, for a guest with 2 virtual nodes, user can specify:
> > +
> > +  vnuma_vdistance = [ [10, 20], [20, 10] ]
> 
> Any guidance on how a user should choose these numbers?
> 

I think using the number from numactl is good enough.

> Do we support a mode where something figures this out based on the
> underlying distances between the pnode to which a vnode is assigned?
> 

Dario is working on that.

> Would a user ever want/need to override such a mapping?
> 

Probably not. But we need to provide a way to override should user want
to do so.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel