Re: [Xen-devel] [PATCH 0/3] Enable L2 Cache Allocation Technology

[Meng Xu <mengxu@xxxxxxxxxxxxx>]

On Thu, Aug 25, 2016 at 1:21 AM, Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx> wrote:
>
> Design document is below:
> =======================================================================
> % Intel L2 Cache Allocation Technology (L2 CAT) Feature
> % Revision 1.0
>
> \clearpage
>
> Hi all,
>
> We plan to bring a new PSR (Platform Shared Resource) feature called
> Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
>
> This is the initial design of L2 CAT. It might be a little long and
> detailed, hope it doesn't matter.
>
> Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> flexible to add new features and fulfill the principle, open for extension
> but closed for modification.
>
> Comments and suggestions are welcome :-)


I have some comments/questions. ;-)


>
>
> # Basics
>
> ---------------- ----------------------------------------------------
>          Status: **Tech Preview**
>
> Architecture(s): Intel x86
>
>    Component(s): Hypervisor, toolstack
>
>        Hardware: Atom codename Goldmont and beyond
> ---------------- ----------------------------------------------------
>
> # Overview
>
> L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> CPU's shared L2 cache based on application priority or Class of Service
> (COS). Each CLOS is configured using capacity bitmasks (CBM) which
> represent cache capacity and indicate the degree of overlap and
> isolation between classes. Once L2 CAT is configured, the processor
> allows access to portions of L2 cache according to the established
> class of service (COS).



The very earlier version of the design at [1] said, "In the initial
implementation, L2 CAT is shown up on Atom codename
Goldmont firstly and there is no platform support both L2 & L3 CAT so far."

I'm wondering if there is any hardware supporting both L2 & L3 CAT now.

[1]http://www.gossamer-threads.com/lists/xen/devel/431142


>
>
> # Technical information
>
> L2 CAT is a member of Intel PSR features and part of CAT, it shares
> some base PSR infrastructure in Xen.
>
> ## Hardware perspective
>
> L2 CAT defines a new range MSRs to assign different L2 cache access
> patterns which are known as CBMs (Capacity BitMask), each CBM is
> associated with a COS.
>
> ```
>
>                         +----------------------------+----------------+
>    IA32_PQR_ASSOC       | MSR (per socket)           |    Address     |
>  +----+---+-------+     +----------------------------+----------------+
>  |    |COS|       |     | IA32_L2_QOS_MASK_0         |     0xD10      |
>  +----+---+-------+     +----------------------------+----------------+
>         └-------------> | ...                        |  ...           |
>                         +----------------------------+----------------+
>                         | IA32_L2_QOS_MASK_n         | 0xD10+n (n<64) |
>                         +----------------------------+----------------+
> ```
>
> When context switch happens, the COS of VCPU is written to per-thread
> MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> according to the corresponding CBM.
>
> ## The relationship between L2 CAT and L3 CAT/CDP
>
> L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
> while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.

This sentence indicates that both L2 and L3 CAT can be enabled at the same time.
How can I know which CPU supports both L2 and L3 CAT?
Is the support architectural or just on several types of CPUs?
IMHO, it will be good to provide a reference about the type of CPUs
that supports L2 only, and that both L2 and L3.

>
> L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
> the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> patterns from L3 cache.
>
> N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> associate register `IA32_PQR_ASSOC`, which means one COS associate to a
> pair of L2 CBM and L3 CBM.
> Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> other PSR features in future). In some cases, a VM is permitted to have a
> COS that is beyond one (or more) of PSR features but within the others.
> For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> limit) since COS 9 is beyond the max COS (8) of L2 CAT.
>
> ## Design Overview
>
> * Core COS/CBM association
>
>   When enforcing L2 CAT, all cores of domains have the same default
>   COS (COS0) which associated to the fully open CBM (all ones bitmask)
>   to access all L2 cache. The default COS is used only in hypervisor
>   and is transparent to tool stack and user.
>
>   System administrator can change PSR allocation policy at runtime by
>   tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
>   to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
>   is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
>   L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
>   things would be easier, one COS corresponds to one L2 CBM.


I felt that this paragraph is kind of confusing. Can system admin
configure both L2 and L3 at the same time for a VM?
What does the 2-tuple, like [L2 CBM, L3 CBM], indicates here?

In addition, does the bits in CBM have to be continuous?

> # User information
>
> * Feature Enabling:
>
>   Add "psr=cat" to boot line parameter to enable all supported level CAT
>   features.
>
> * xl interfaces:
>
>   1. `psr-cat-show [OPTIONS] domain-id`:
>
>      Show domain L2 or L3 CAT CBM.
>
>      New option `-l` is added.
>      `-l2`: Specify cbm for L2 cache.
>      `-l3`: Specify cbm for L3 cache.
>
>      If neither `-l2` nor `-l3` is given, LLC (Last Level Cache) is
>      default behavior.

Does it mean the LLC CAT setting will be reset or just kept to the
latest configuration?

>
>   2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
>
>      Set domain L2 or L3 CBM.
>
>      New option `-l` is added.
>      `-l2`: Specify cbm for L2 cache.
>      `-l3`: Specify cbm for L3 cache.
>
>      If neither `-l2` nor `-l3` is given, LLC (Last Level Cache) is
>      default behavior. Since there is no CDP support on L2 cache, so if
>      `-l2`, `--code` or `--data` are both given, error message shows.
>
>   3. `psr-hwinfo [OPTIONS]`:
>
>      Show L2 & L3 CAT HW informations on every socket.
>
> # References
>
> [Intel® 64 and IA-32 Architectures Software Developer 
> Manuals](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)

Would you mind provide the chapter for the L2 CAT and the version of
the SDM you are referring?
It makes people's life easier to find the reference. ;-)

>
> # History
>
> ------------------------------------------------------------------------
> Date       Revision Version  Notes
> ---------- -------- -------- -------------------------------------------
> 2016-08-12 1.0      Xen 4.7  Initial design

It seems there is a RFC design in May this year:
http://www.gossamer-threads.com/lists/xen/devel/431142

Shouldn't it be counted?

Thanks,

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel