Re: [Xen-devel] [PATCH] x86/pt: skip setup of posted format IRTE when gvec is 0

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Tue, Apr 30, 2019 at 05:48:14PM +0100, Andrew Cooper wrote:
> On 30/04/2019 17:24, Chao Gao wrote:
> > On Tue, Apr 30, 2019 at 11:08:54AM +0200, Roger Pau Monné wrote:
> >> On Tue, Apr 30, 2019 at 01:19:19PM +0800, Chao Gao wrote:
> >>> When testing with an UP guest with a pass-thru device with vt-d pi
> >>> enabled in host, we observed that guest couldn't receive interrupts
> >>> from that pass-thru device. Dumping IRTE, we found the corresponding
> >>> IRTE is set to posted format with "vector" field as 0.
> >>>
> >>> We would fall into this issue when guest used the pirq format of MSI
> >> I think the above sentence is a bit confusing. I would rather say that
> >> the guest is routing interrupts from passthrough devices over event
> >> channels using pirqs.
> > Yes. It is better than it was.
> >
> >>> (see the comment xen_msi_compose_msg() in linux kernel). As 'dest_id'
> >>> is repurposed, skip migration which is based on 'dest_id'.
> >> Like Jan, I wonder why the device model (I assume QEMU) issues a
> >> XEN_DOMCTL_bind_pt_irq hypercall when the guest is attempting to route
> >> this interrupts over an event channel, attempting to bind it to a
> >> vector seems like a bug in the device model.
> >>
> >> I would also consider whether it would make sense to not expose the
> >> XENFEAT_hvm_pirqs feature when doing passthrough if posted interrupts
> >> are supported, performance wise it's better to use posted interrupts
> >> rather than routing them over event channels (which requires Xen
> >> interaction).
> > It is reasonable. But I think currently guest can add "xen_nopv" to its
> > kernel boot options to avoid using evtchn. There might be some cases
> > even with pass-thru devices (such as, guest uses polling mode
> > driver for pass-thru devices), we care more about the efficiency of
> > delivering virtual interrupts. So a separate option for evtchn in guest
> > kernel seems like a better choice.
> 
> This is a festering swamp, and is going to need some careful
> architectural work to untangle.
> 
> At the moment, guests which are xen-aware will try to use event channels
> for everything.  I'm pretty sure we've got some ABI incompatibilities
> here with hardware APIC acceleration, and I still have yet(!) to
> diagnose the underlying problem which is preventing XenServer from
> enabling hardware APIC acceleration.  (VMs still intermittently wedge on
> migrate with what appears to be lost interrupt.)

My opinion is that XENFEAT_hvm_pirqs should be removed:

 - It abuses of a hardware interface to pass Xen specific information.
   When routing pirqs over event channels for HVM the pirq is passed
   on the MSI address field, hijacking the native dest_id field.
 - Cannot be used with posted interrupts or APICV.

Iff routing physical interrupts over event channels is still a useful
thing for HVM, it should be done using hypercalls, like it's done on a
PV dom0.

> 
> The legacy HVM callback via single vector is incompatible with hardware
> APIC acceleration, because it exists specifically to skip going through
> the IRR.  I can't think a proper solution to this, other than crashing
> the guest when it tries to set such a situation up.
> 
> From an "architecturally ideal" point of view, we obviously want to be
> using APICV and/or Posted interrupt support whenever possible, and be
> using these methods in preference to event channels.
> 
> Injecting interrupts (including signalling an event upcall) using the
> PIR is more efficient than other means, so long as we can ensure that
> guest will handle the vector like any other edge triggered interrupt,
> and ack it at the LAPIC.  (This is where the legacy callback via method
> breaks down, as it was intentionally designed to be used without an ack
> at the LAPIC.)

IIRC there's already a correct way to setup an event channel upcall
against a vector using HVMOP_set_evtchn_upcall_vector which requires
an APIC ack. Now the issue is switching existing users to this new
callback setup mechanism.

> As soon as we get to device interrupts, posted is definitely the way to
> go, especially as the guest should not be aware of the routing behind
> the scenes in the first place.
> 
> To be clear, Chao - I'm not asking you to fix this.  This is a decade of
> technical debt which has been steadily growing, and it needs to start
> with a coherent plan what the current state of play is, and how we'd
> like to proceed.

I agree. I've suggested a possible fix for the problem at hand, but as
stated above I think XENFEAT_hvm_pirqs should be removed.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel