Linux PV domU with >1 vCPU never resumes after xl save/restore

[George Dunlap <gwd@xxxxxxxxxxxxxx>]

Hello,

Saving and restoring a multi-vcpu PV guest appears to have been broken in Linux for some time (observed 6.6.56 and 6.12.86).  Report below from Claude Fable; I've independently verified the behavior on vanilla Linux 6.6.56.  Claude seems to think it's a bug in Linux.

Gitlab CI seems to only run with vcpus=1, which is why it didn't notice.

George Dunlap

Freelance Xen consultant

https://www.laleolanguage.com/consulting

8<----

A PV guest with more than one vCPU survives `xl save`, but after
`xl restore` it never comes back: the kernel wedges mid-resume, before
xenbus reconnect, so all frontends stay disconnected (netfront frontend
state remains XenbusStateInitialising, backend InitWait; vif shows
NO-CARRIER in dom0) and the guest is unreachable indefinitely. With
vcpus=1 the same guest/image/kernel resumes cleanly, PVH SMP
save/restore is fine, and the suspend-cancel path (a failed `xl save`
resuming the domain in place) is also fine.

Reproduced with:
 - Debian trixie kernel 6.12.86+deb13-amd64
 - the Xen-project CI test-artifacts kernel, vanilla 6.6.56
 (identical signature on both, so not a 6.12 regression; at least the
 6.6..6.12 LTS span is affected)
Host: x86-64, Xen master/staging (4.23-unstable); also reproduced on an
older commit, so the Xen version does not appear relevant. Plain
`xl save` + `xl restore` of an idle 4-vCPU, 2G PV domU, direct kernel
boot, xvda file-backed disk, one vif. 100% reproducible.

What the resume looks like (full logs available):

 - Capturing the console across a paused restore (`xl restore -p`,
   attach console, unpause) shows all secondary vCPUs immediately
   splatting:

     WARNING: CPU: 1 PID: 0 at kernel/time/timekeeping.c:747 ktime_get+0xa9/0xd0
     ...
      tick_nohz_idle_enter
      do_idle
      cpu_startup_entry
      cpu_bringup_and_idle
      asm_cpu_bringup_and_idle

   i.e. the idle task entering nohz while timekeeping is still
   suspended — with printk timestamps taken from the *uncorrected*
   clock (pre-suspend time + the save/restore wall-clock gap), while
   CPU0's own subsequent resume messages ("Grant tables using version 1
   layout", from gnttab_resume() inside xen_suspend()) carry the
   *corrected*, earlier timestamp. The secondaries therefore left the
   stop_machine corral before CPU0's post-suspend work inside
   xen_suspend() had run, which the multi_cpu_stop state machine is
   supposed to make impossible.

 - xenctx on the restored-but-still-paused domain shows the vCPU
   contexts are restored faithfully (RIP-for-RIP identical to a probe
   taken at the suspend point: vCPU0 inside the suspend hypercall stub,
   the secondaries inside the multi_cpu_stop corral loop). The
   toolstack is delivering exactly what was saved; the wedge develops
   after unpause, guest-side.

 - End state, stable forever after: vCPU0 spins at 100% inside the
   multi_cpu_stop corral code (per xenctx; `xl vcpu-list` shows r--
   accumulating time), while the secondary vCPUs sit blocked in
   SCHED_block on their idle-task stacks. Because stop_machine() never
   completes, do_suspend() never reaches xen_arch_resume() (so the
   secondaries' local ticks, suspended by xen_arch_suspend() before the
   corral, are never resumed — nothing will ever wake them) nor
   xs_resume()/dpm_resume_*() (so xenbus frontends never reconnect).

A speculative note on the trigger, from reading the 6.12 code — treat
as unverified: xen_vcpu_restore() (called from xen_pv_post_suspend()
while the secondaries are mid-corral with virtual interrupts masked)
does VCPUOP_down, re-registers vcpu_info via xen_vcpu_setup_restore(),
then VCPUOP_up on each secondary. If the re-registration ends up with
evtchn_upcall_mask clear in the newly registered vcpu_info, the vCPU
comes back up with an unexpected upcall window mid-corral; stray
exc_xen_hypervisor_callback frames in the secondaries' backtraces are
consistent with that. I stopped root-causing at this point.

Happy to provide the full console logs, xenctx dumps at
suspend/restored-paused/wedged, and the reproduction scripts, or to
test patches.