Re: Improper handling of fast-resumed domains by O/Cxenstored

[Jürgen Groß <jgross@xxxxxxxx>]

On 07.07.26 15:07, Andrii Sultanov wrote:
> Both Oxenstored and Cxenstored don't properly handle a shutdown domain
> (even if shutdown_code == SHUTDOWN_suspend) coming back because of
> xc_domain_resume(fast=1).
>
> Cxenstored correctly sends @introduceDomain watch events on the
> initial INTRODUCE call:
>
>       IN 0x1f78610 20260619 15:45:37 INTRODUCE (1 1044476 2 )
>       CREATE connection 0x1f75870
>       OUT 0x1f76f00 20260619 15:45:37 WATCH_EVENT (@introduceDomain domlist )
>       OUT 0x1f76ce0 20260619 15:45:37 WATCH_EVENT (@introduceDomain
> squeezed )
>       OUT 0x1f77c40 20260619 15:45:37 WATCH_EVENT (@introduceDomain  )
>       OUT 0x1f78610 20260619 15:45:37 WATCH_EVENT (@introduceDomain  )
>       OUT 0x1f78610 20260619 15:45:37 INTRODUCE (OK )
>
> And it sends @releaseDomain when it notices dominfo.shutdown (without
> checking for shutdown_code):
>
>       OUT 0x1f76f00 20260619 15:46:56 WATCH_EVENT (@releaseDomain domlist )
>       OUT 0x1f76ce0 20260619 15:46:56 WATCH_EVENT (@releaseDomain squeezed )
>       OUT 0x1f77c40 20260619 15:46:56 WATCH_EVENT (@releaseDomain )
>       OUT 0x1f78610 20260619 15:46:56 WATCH_EVENT (@releaseDomain )
>
> But after fast resume, even though INTRODUCE returns successfully,
> @introduceDomain watch events are not re-sent:
>
>       IN 0x1f78610 20260619 15:47:06 INTRODUCE (1 1044476 2 )
>       <----- no watch events
>       OUT 0x1f78610 20260619 15:47:06 INTRODUCE (OK )
>
> Oxenstored likewise avoids sending @introduceDomain for "re-connecting"
> domains (which is what it thinks a domain coming back like this looks like).
>
> RESUME does not re-send @introduceDomain either, it just marks the domain as
> live internally in Cxenstored and is a no-op for Oxenstored.
>
> This missing notification leads to all kinds of issues since the toolstacks
> don't notice the resumed domain is back (at least on the XAPI side).
>
> Is this a real issue or is the toolstack holding xenstored wrong here?
>
> If it is, I see two possible options for handling this:
>
> 1) Don't consider SHUTDOWN_suspend domains as dead, only drop them when they
>      are definitively gone. This avoids the @releaseDomain notification
>      and hence doesn't require another @introduceDomain
>
> 2) Do actually send out @introduceDomain for thought-to-be-shutdown domains
>      on INTRODUCE
>
> Thoughts?
I believe 1) is the way to go here.

In fact this is a hypervisor issue. In __domain_finalise_shutdown() the call
of domain_changed_state() should only happen if send_global_virq(VIRQ_DOM_EXC)
is done.

Could you please test whether the following diff is helping?

diff --git a/xen/common/domain.c b/xen/common/domain.c
index 53ac1d6c40..a4d9095c20 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -286,11 +286,12 @@ static void __domain_finalise_shutdown(struct domain *d)
             return;

     d->is_shut_down = 1;
-    domain_changed_state(d);
     if ( (d->shutdown_code == SHUTDOWN_suspend) && d->suspend_evtchn )
         evtchn_send(d, d->suspend_evtchn);
-    else
+    else {
+        domain_changed_state(d);
         send_global_virq(VIRQ_DOM_EXC);
+    }
 }

 static void vcpu_check_shutdown(struct vcpu *v)



Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature