[Xen-devel] [RFC PATCH 00/24] [FOR 4.9] arm64: Dom0 ITS emulation

[Andre Przywara <andre.przywara@xxxxxxx>]

Hi,

apologies for sending this series, which due to its early status is
now targeting Xen 4.9, still before the 4.8 hard feature freeze, but
there seems to be some interested parties and I wanted to get the
discussion started on this.

This series introduces ARM GICv3 ITS emulation, for now restricted to
Dom0 only. The ITS is an interrupt controller widget providing a
sophisticated way to deal with MSIs in a scalable manner.
For hardware which relies on the ITS to provide interrupts for its
peripherals this code is needed to get a machine booted into Dom0 at all.
ITS emulation for DomUs is only really useful with PCI passthrough,
which is not yet available for ARM. It is expected that this feature
will be co-developed with the ITS DomU code.

This implementation is totally independent from earlier submissons and
tries to provide a new approach:

* The current GIC code statically allocates structures for each supported
IRQ (both for the host and the guest), which due to the potentially
millions of LPI interrupts is not feasible to copy for the ITS.
So we refrain from introducing the ITS as a first class Xen interrupt
controller, also we don't hold struct irq_desc's or struct pending_irq's
for each possible LPI.
Fortunately LPIs are only interesting to guests, so we get away with
storing only the virtual IRQ number and the guest VCPU for each allocated
host LPI, which can be stashed into one uint64_t. This data is stored in
a two-level table, which is both memory efficient and quick to access.
We hook into the existing IRQ handling and VGIC code to avoid accessing
the normal structures, providing alternative methods for getting the
needed information (priority, is enabled?) for LPIs.
For interrupts which are queued to or are actually in a guest we
allocate struct pending_irq's on demand. As it is expected that only a
very small number of interrupts is ever on a VCPU at the same time, this
seems like the best approach. For now allocated structs are re-used and
held in a linked list.

* On the guest side we (later will) have to deal with malicious guests
trying to hog Xen with mapping requests for a lot of LPIs, for instance.
As the ITS actually uses system memory for storing status information,
we use this memory (which the guest has to provide) to naturally limit
a guest. For those tables which are page sized (devices, collections (CPUs),
LPI properties) we map those pages into Xen, so we can easily access
them from the virtual GIC code.
Unfortunately the actual interrupt mapping tables are not necessarily
page aligned, also can be much smaller than a page, so mapping all of
them permanently is fiddly. As ITS commands in need to iterate those
tables are pretty rare after all, we for now map them on demand upon
emulating a virtual ITS command.

* At least for the Dom0 use case we need to pass some virtual ITS
commands on to the hardware, namely device and interrupt mapping requests.
As we have to deal with command emulation synchronously, this could lead
to situations where the host command queue gets congested and CPUs stall
on this. For now we assume that Dom0 (as being non-malicious) is not
affected by this, so we allow this.
DomUs will later use PCI passthrough, so we can do the device and also
IRQ mapping upon domain creation time, obviating ITS command passthrough
for DomUs during their runtime at all.

This series is an early draft, with some known and many unknown issues.
I made ITS support a Kconfig option, also it is only supported on arm64.
This leads to some hideous constructs like an #ifdef'ed header file with
empty function stubs, but I guess we can clean this up later in the
upstreaming process.
Also I am not sure the host ITS and LPI initialization code is correct,
especially when it comes to the subtle differences between ITS and LPIs
enabling and initialization. Affinity handling is only rudimentarily
implemented at this point. Also locking some of the shared data structures
isn't fully implemented yet, partly because Xen's lack of mutexes, RCUs
and preemption requiring some more clever solutions than my Linux
experiences would easily provide offhand.

So for now I am mostly interested in feedback on the general architecture
approach. Please comment on anything that looks suspicious or not
sustainable.
Oh, and apologies in advance for any tabs and missing "spaces in
if-statements" that slipped through my (unfortunately only manual) QA
process.

For now this code happens to boot Dom0 on an ARM fast model with ITS
support. I haven't had the chance to get hold of a Xen supported hardware
platform with an ITS yet, so I expect some surprises when this code sees
real hardware for the first time ;-)
That being said any testing and feedback is warmly welcomed!

The code can also be found on the its/rfc branch here:
git://linux-arm.org/xen-ap.git
http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/its/rfc

Cheers,
Andre

Andre Przywara (24):
  ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
  ARM: GICv3: allocate LPI pending and property table
  ARM: GICv3 ITS: allocate device and collection table
  ARM: GICv3 ITS: map ITS command buffer
  ARM: GICv3 ITS: introduce ITS command handling
  ARM: GICv3 ITS: introduce host LPI array
  ARM: GICv3 ITS: introduce device mapping
  ARM: GICv3: introduce separate pending_irq structs for LPIs
  ARM: GICv3: forward pending LPIs to guests
  ARM: GICv3: enable ITS and LPIs on the host
  ARM: vGICv3: handle virtual LPI pending and property tables
  ARM: vGICv3: introduce basic ITS emulation bits
  ARM: vITS: handle CLEAR command
  ARM: vITS: handle INT command
  ARM: vITS: handle MAPC command
  ARM: vITS: handle MAPD command
  ARM: vITS: handle MAPTI command
  ARM: vITS: handle MOVI command
  ARM: vITS: handle DISCARD command
  ARM: vITS: handle INV command
  ARM: vITS: handle INVALL command
  ARM: vITS: create and initialize virtual ITSes for Dom0
  ARM: vITS: create ITS subnodes for Dom0 DT
  ARM: vGIC: advertising LPI support

 xen/arch/arm/Kconfig              |  11 +
 xen/arch/arm/Makefile             |   2 +
 xen/arch/arm/efi/efi-boot.h       |   1 -
 xen/arch/arm/gic-its.c            | 861 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3.c             |  78 +++-
 xen/arch/arm/gic.c                |   9 +-
 xen/arch/arm/vgic-its.c           | 861 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic-v3.c            | 240 +++++++++--
 xen/arch/arm/vgic.c               |  60 ++-
 xen/include/asm-arm/cache.h       |   4 +
 xen/include/asm-arm/domain.h      |   9 +-
 xen/include/asm-arm/gic-its.h     | 246 +++++++++++
 xen/include/asm-arm/gic_v3_defs.h |  67 ++-
 xen/include/asm-arm/irq.h         |   8 +
 xen/include/asm-arm/vgic.h        |  12 +
 15 files changed, 2432 insertions(+), 37 deletions(-)
 create mode 100644 xen/arch/arm/gic-its.c
 create mode 100644 xen/arch/arm/vgic-its.c
 create mode 100644 xen/include/asm-arm/gic-its.h

-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel