Re: [Xen-devel] [RFC] ARM: New (Xen) VGIC design document

[Christoffer Dall <christoffer.dall@xxxxxxxxxx>]

On Wed, Nov 1, 2017 at 9:58 AM, Stefano Stabellini
<sstabellini@xxxxxxxxxx> wrote:

[....]

>
>> ### List register management
>>
>> A list register (LR) holds the state of a virtual interrupt, which will
>> be used by the GIC hardware to simulate an IRQ life cycle for a guest.
>> Each GIC hardware implementation can choose to implement a number of LRs,
>> having four of them seems to be a common value. This design here does not
>> try to manage the LRs very cleverly, instead on every guest exit every LR
>> in use will be synced to the emulated state, then cleared. Upon guest entry
>> the top priority virtual IRQs will be inserted into the LRs. If there are
>> more pending or active IRQs than list registers, the GIC management IRQ
>> will be configured to notify the hypervisor of a free LR (once the guest
>> has EOIed one IRQ). This will trigger a normal exit, which will go through
>> the normal cleanup/repopulate scheme, possibly now queuing the leftover
>> interrupt(s).
>> To facilitate quick guest exit and entry times, the VGIC maintains the list
>> of pending or active interrupts (ap\_list) sorted by their priority. Active
>> interrupts always go first on the list, since a guest and the hardware GIC
>> expect those to stay until they have been explicitly deactivated. Failure
>> in keeping active IRQs around will result in error conditions in the GIC.
>> The second sort criteria for the ap\_list is their priority, so higher
>> priority pending interrupt always go first into the LRs.
>
> The suggestion of using this model in Xen was made in the past already.
> I always objected for the reason that we don't actually know how many
> LRs the hardware provides, potentially very many, and it is expensive
> and needless to read/write them all every time on entry/exit.
>
> I would prefer to avoid that, but I'll be honest: I can be convinced
> that that model of handling LRs is so much simpler that it is worth it.
> I am more concerned about the future maintainance of a separate new
> driver developed elsewhere.

[Having just spent a fair amount of time optimizing KVM/ARM and
measuring GIC interaction, I'll comment on this and leave it up to
Andre to drive the rest of the discussion].

In KVM we currently only ever touch an LR when we absolutely have to.
For example, if there are no interrupts, we do not touch an LR.

When you do have an interrupt in flight, and have programmed one or
more LRs, you have to either read back that LR, or read one of the
status registers to figure out if the interrupt has become inactive
(and should potentially be injected again).  I measured both on KVM
for various workloads and it was faster to never read the status
registers, but simply read back the LRs that were in use when entering
the guest.

You can potentially micro-optimize slightly by remembering the exit
value of an LR (and not clearing it on guest exit), but you have to
pay the cost in terms of additional logic during VCPU migration and
when you enter a VM again, maintaining a mapping of the LR and the
virtual state, to avoid rewriting the same value to the LR again.  We
tried that in KVM and could not measure any benefit using either a
pinned or oversubscribed workload; I speculate that the number of
times you exit with unprocessed interrupts in the LRs is extremely
rare.

In terms of the number of LRs, I stil haven't seen an implementation
with anything else than 4 LRs.

-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel