Re: [Xen-devel] [PATCH] x86/pt: skip setup of posted format IRTE when gvec is 0

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 30/04/2019 17:24, Chao Gao wrote:
> On Tue, Apr 30, 2019 at 11:08:54AM +0200, Roger Pau Monné wrote:
>> On Tue, Apr 30, 2019 at 01:19:19PM +0800, Chao Gao wrote:
>>> When testing with an UP guest with a pass-thru device with vt-d pi
>>> enabled in host, we observed that guest couldn't receive interrupts
>>> from that pass-thru device. Dumping IRTE, we found the corresponding
>>> IRTE is set to posted format with "vector" field as 0.
>>>
>>> We would fall into this issue when guest used the pirq format of MSI
>> I think the above sentence is a bit confusing. I would rather say that
>> the guest is routing interrupts from passthrough devices over event
>> channels using pirqs.
> Yes. It is better than it was.
>
>>> (see the comment xen_msi_compose_msg() in linux kernel). As 'dest_id'
>>> is repurposed, skip migration which is based on 'dest_id'.
>> Like Jan, I wonder why the device model (I assume QEMU) issues a
>> XEN_DOMCTL_bind_pt_irq hypercall when the guest is attempting to route
>> this interrupts over an event channel, attempting to bind it to a
>> vector seems like a bug in the device model.
>>
>> I would also consider whether it would make sense to not expose the
>> XENFEAT_hvm_pirqs feature when doing passthrough if posted interrupts
>> are supported, performance wise it's better to use posted interrupts
>> rather than routing them over event channels (which requires Xen
>> interaction).
> It is reasonable. But I think currently guest can add "xen_nopv" to its
> kernel boot options to avoid using evtchn. There might be some cases
> even with pass-thru devices (such as, guest uses polling mode
> driver for pass-thru devices), we care more about the efficiency of
> delivering virtual interrupts. So a separate option for evtchn in guest
> kernel seems like a better choice.

This is a festering swamp, and is going to need some careful
architectural work to untangle.

At the moment, guests which are xen-aware will try to use event channels
for everything.  I'm pretty sure we've got some ABI incompatibilities
here with hardware APIC acceleration, and I still have yet(!) to
diagnose the underlying problem which is preventing XenServer from
enabling hardware APIC acceleration.  (VMs still intermittently wedge on
migrate with what appears to be lost interrupt.)

The legacy HVM callback via single vector is incompatible with hardware
APIC acceleration, because it exists specifically to skip going through
the IRR.  I can't think a proper solution to this, other than crashing
the guest when it tries to set such a situation up.

From an "architecturally ideal" point of view, we obviously want to be
using APICV and/or Posted interrupt support whenever possible, and be
using these methods in preference to event channels.

Injecting interrupts (including signalling an event upcall) using the
PIR is more efficient than other means, so long as we can ensure that
guest will handle the vector like any other edge triggered interrupt,
and ack it at the LAPIC.  (This is where the legacy callback via method
breaks down, as it was intentionally designed to be used without an ack
at the LAPIC.)

As soon as we get to device interrupts, posted is definitely the way to
go, especially as the guest should not be aware of the routing behind
the scenes in the first place.

To be clear, Chao - I'm not asking you to fix this.  This is a decade of
technical debt which has been steadily growing, and it needs to start
with a coherent plan what the current state of play is, and how we'd
like to proceed.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel