[PATCH 4/6] vmx/apicv: Fully deassign the PI hooks when the last device is removed

[Teddy Astie <teddy.astie@xxxxxxxxxx>]

vmx_pi_hooks_deassign() has been keeping the 'switch_to' hook installed
forever since 7c00de17562e, for it to clear SN when a vCPU which was
descheduled with SN set runs again.  It also has to set ON if anything
got posted to the PIR while SN was set, as otherwise these vectors
would stay parked in there.

The domain is paused at that point, so there's no need to defer any of
this to the next context switch: clear SN and set ON as needed right
away, and drop PI_CSW_TO alongside PI_CSW_FROM.  Without the hook, NDST
isn't kept up to date anymore, so reset it to APIC_INVALID_DEST for
vmx_pi_hooks_assign() to set it up again upon a later assignment, as it
does for a freshly created vCPU.

While at it, also wake up blocked vCPUs with ON set:
vmx_pi_unblock_vcpu() resets NV to posted_intr_vector before taking the
vCPU off the blocking list, so a posting landing in between is only
notified to the pCPU, and nothing would wake the vCPU.

Fixes: 7c00de17562e ("VMX: permanently assign PI hook vmx_pi_switch_to()")
Assisted-by: Claude Opus 5.5
Signed-off-by: Teddy Astie <teddy.astie@xxxxxxxxxx>

---
We're removing the comment that the previous commit fixed.
I'm not sure if that would be preferable to merge this one
with the previous one.
---
 xen/arch/x86/hvm/vmx/vmx.c | 46 ++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index 3d77d77534..6db2529730 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -394,25 +394,43 @@ void vmx_pi_hooks_deassign(struct domain *d)
     ASSERT(current->domain != d);
     domain_pause(d);
 
-    /*
-     * Note that we keep the 'switch_to' hook.  Any vCPU which isn't blocked
-     * has just been descheduled by domain_pause() with the 'switch_from'
-     * hook still in place, and hence has 'SN' set.  Interrupts posted by the
-     * IOMMU while 'SN' is set only have their vector recorded in the PIR,
-     * without setting 'ON'.  vmx_pi_switch_to() clears 'SN' and sets 'ON'
-     * if the PIR isn't empty, so that such vectors get delivered once the
-     * vCPU runs again.  It also keeps 'NDST' up to date, which
-     * vmx_pi_hooks_assign() relies upon should a device get assigned again.
-     *
-     * Once 'SN' is clear it stays clear, as nothing sets it anymore, and the
-     * hook is redundant from then on.
-     */
     d->arch.hvm.pi_ops.vcpu_block = NULL;
-    d->arch.hvm.pi_ops.flags = PI_CSW_TO;
+    d->arch.hvm.pi_ops.flags = 0;
 
     for_each_vcpu ( d, v )
+    {
+        struct pi_desc *pi_desc = &v->arch.hvm.vmx.pi_desc;
+
         vmx_pi_unblock_vcpu(v);
 
+        /*
+         * Any vCPU which isn't blocked has just been descheduled by
+         * domain_pause() with the 'switch_from' hook still in place, and
+         * hence has 'SN' set.  Nothing sets it anymore from now on, so do
+         * what vmx_pi_switch_to() would have done when the vCPU next runs:
+         * clear 'SN', and set 'ON' if anything got posted in the meantime.
+         */
+        pi_clear_sn(pi_desc);
+        if ( !pi_is_pir_empty(pi_desc) )
+            pi_set_on(pi_desc);
+
+        /*
+         * A blocked vCPU may have had an interrupt posted after
+         * vmx_pi_unblock_vcpu() reset 'NV' to posted_intr_vector, and, now
+         * that it is off the blocking list, nobody will wake it up.  The
+         * domain is paused, so a spurious wakeup is harmless.
+         */
+        if ( pi_test_on(pi_desc) )
+            vcpu_unblock(v);
+
+        /*
+         * Reset 'NDST' to its initial value, for vmx_pi_hooks_assign() to
+         * set it up again if a device gets assigned later on: without the
+         * 'switch_to' hook it isn't kept up to date anymore.
+         */
+        write_atomic(&pi_desc->ndst, APIC_INVALID_DEST);
+    }
+
     domain_unpause(d);
 }
 
-- 
2.56.0



--
Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech