Re: [Xen-devel] [RFC v2] xen/arm: Suspend to RAM Support in Xen for ARM

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 22 Dec 2017, Mirela Simonovic wrote:
> This document contains our design specification for "suspend to RAM"
> support for ARM in Xen. It covers the basic suspend to RAM mechanism
> based on ARM PSCI standard, that would allow individual guests and
> Xen itself to suspend/resume.
> 
> We would appreciate your feedback.
> 
> Signed-off-by: Mirela Simonovic <mirela.simonovic@xxxxxxxxxx>

Sounds good to me

Acked-by: Stefano Stabellini <sstabellini@xxxxxxxxxx>


> ---
> v2:
> -Improved specification according to comments
> -Added more implementation details
> -Incremented revision number to 1.1
> ---
>  docs/misc/arm/suspend-to-ram.txt | 266 
> +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 266 insertions(+)
>  create mode 100644 docs/misc/arm/suspend-to-ram.txt
> 
> diff --git a/docs/misc/arm/suspend-to-ram.txt 
> b/docs/misc/arm/suspend-to-ram.txt
> new file mode 100644
> index 0000000000..6e8f10d1ce
> --- /dev/null
> +++ b/docs/misc/arm/suspend-to-ram.txt
> @@ -0,0 +1,266 @@
> +% Suspend to RAM Support in Xen for ARM
> +% Revision 1.1
> +
> +========
> +Overview
> +========
> +
> +Suspend to RAM (in the following text 'suspend') for ARM in Xen should be
> +coordinated using ARM PSCI standard [1].
> +
> +Ideally, EL1/2 should suspend in the following order:
> +1) Unprivileged guests (DomUs) suspend
> +2) Privileged guest (Dom0) suspends
> +3) Xen suspends
> +
> +However, suspending unprivileged guests is not mandatory for suspending
> +Dom0 and Xen. System suspend initiated by Dom0 (step 2) is considered to be 
> an
> +ultimate decision to suspend the physical machine. Suspending of Xen (step 3)
> +is triggered whenever Dom0 completes suspend. Xen suspend leads to the full
> +suspend of EL2.
> +
> +If an unprivileged guest is not suspended at the moment when Dom0 initiates
> +its own suspend, the guest will be paused on Xen's suspend and unpaused on
> +Xen's resume. That way, a guest which doesn't have power management support
> +cannot prevent the physical system from suspending when the decision to 
> suspend
> +is made by privileged software (Dom0).
> +
> +Each guest in the system is responsible for suspending the devices it owns.
> +If a guest does not suspend a device, the device will continue to operate as
> +it is configured at the moment when the system suspends. If a device triggers
> +an interrupt while the physical system is suspended, the system will resume.
> +
> +It is recommended for an unprivileged guest to participate in power 
> management
> +in the following scenario:
> +Assume unprivileged guest owns a device which will trigger interrupt at some
> +point. This interrupt will wake-up the system. If such a behavior is not 
> wanted,
> +coordination between Dom0 and the guest is required in order to inform the 
> guest
> +about the intended physical system suspend. Then, the guest will have a 
> chance
> +to suspend the device or respond to the request in an abort fashion.
> +
> +Since this proposal is focused on implementing PSCI-based suspend mechanisms 
> in
> +Xen, communication with or among the guests is not covered by this document.
> +The order of suspending the guests is assumed to be guaranteed by the 
> software
> +running in EL1.
> +
> +This document covers the following:
> +1) Mechanism for suspending/resuming a guest:
> +     1.1) Suspend is initiated by the guest
> +     1.2) Resume is initiated by a device interrupt
> +2) Mechanism for pausing/unpausing running guests when Dom0 suspends
> +3) Mechanism for suspending/resuming Xen when Dom0 completes suspend
> +4) Resuming from any state on a wake-up event (device interrupt):
> +     4.1) Resume DomU on wake-up event when Dom0 is still running
> +     4.2) Resume DomU on wake-up event when Xen is suspended
> +     4.3) Resume Dom0 on wake-up event
> +
> +Mechanisms enumerated above will allow different kind of policies and
> +coordination among guests to be implemented in EL1. That is out of the scope 
> of
> +this document.
> +
> +-----------------
> +Suspending Guests
> +-----------------
> +
> +Suspend procedure for a guest consists of the following:
> +1) Suspending devices
> +2) Suspending non-boot CPUs (based on hotplug/PSCI)
> +3) System suspend, performed by the boot CPU
> +
> +Each guest should suspend the devices it owns just like it would when running
> +without Xen.
> +
> +Guests should suspend their non-boot vCPUs using the hotplug mechanism.
> +Virtual CPUs should be put offline using the already implemented PSCI 
> vCPU_OFF
> +call (prefix 'v' is added to distinguish PSCI calls made by guests to Xen, 
> which
> +affect virtual machines; as opposed to PSCI calls made by Xen to the EL3, 
> which
> +can affect power state of the physical machine).
> +
> +After suspending its non-boot vCPUs a guest should finalize the suspend by
> +making the vSYSTEM_SUSPEND PSCI call. The resume address is specified by the
> +guest via the vSYSTEM_SUSPEND entry_point_address argument. The 
> vSYSTEM_SUSPEND
> +call is currently not implemented in Xen.
> +
> +It is expected that a guest leaves enabled all interrupts that should wake it
> +up. Other interrupts should be disabled by the guest prior to calling
> +vSYSTEM_SUSPEND.
> +
> +After an unprivileged guest suspends, Xen will not suspend. Xen would suspend
> +only after the Dom0 completes the system suspend.
> +
> +--------------
> +Suspending Xen
> +--------------
> +
> +Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND call
> +from the last running guest (Dom0). At that moment all physical CPUs are 
> still
> +online (taking offline a vCPU or suspending a VM does not affect physical 
> CPUs).
> +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI call
> +to EL3. The CPU_OFF PSCI function is currently not implemented in Xen.
> +
> +After putting offline the non-boot cores Xen must save the context and 
> finalize
> +suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to EL3.
> +The resume point of Xen is specified by the entry_point_address argument of 
> the
> +SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and context saving is not
> +implemented in Xen for ARM today.
> +
> +------------
> +Resuming Xen
> +------------
> +
> +Xen must be resumed prior to any software running in EL1. Starting from the
> +resume point, Xen should restore the context and resume Dom0. Dom0 shall 
> always
> +be resumed whenever Xen resumes.
> +
> +---------------
> +Resuming Guests
> +---------------
> +
> +Resume of the privileged guest (Dom0) is always following the Xen resume.
> +
> +An unprivileged guest shall resume once a device it owns triggers a wake-up
> +interrupt, regardless of whether Xen was suspended when the wake-up interrupt
> +was triggered. If Xen was suspended, it is assumed that Dom0 will be running
> +before the DomU guest starts to resume. The synchronization mechanism to
> +enforce the assumed condition is TBD.
> +
> +If the ARM's GIC was powered down after the ARM subsystem suspended, it is
> +assumed that Xen needs to restore the GIC interface for a VM prior to handing
> +over control to the guest. However, the guest should restore its own context
> +upon entering the resume point, just like it would when running without Xen.
> +
> +===============
> +Implementation
> +===============
> +
> +--------
> +Overview
> +--------
> +
> +In order to enable the suspend/resume of VMs and Xen itself, the following 
> PSCI
> +calls have to be implemented and integrated in Xen:
> +1) vSYSTEM_SUSPEND
> +2) CPU_OFF
> +3) SYSTEM_SUSPEND
> +
> +In addition, the following have to be implemented:
> +* Suspend/resume vCPU (triggered by vSYSTEM_SUSPEND call)
> +* Suspend/resume Xen (triggered by vSYSTEM_SUSPEND called by Dom0), 
> including:
> +     * Disable wathdog on suspend, enable it on resume
> +     * Pause domains on suspend, unpause them on resume
> +     * Disable non-boot pCPUs on suspend, enable them on resume
> +     * Save/restore of GIC configuration
> +     * Suspend/resume timer
> +     * Save/restore of EL2 context
> +     * Implement resume entry point in Xen, including MMU configuration
> +
> +Implementation details are provided in the sections below. Function names and
> +paths used below are consistent within the document but may not always match 
> the
> +names used in future implementation. Existing functions and paths are named 
> as
> +in Xen source tree.
> +
> +-------------------------------------
> +Suspend/Resume Implementation Details
> +-------------------------------------
> +
> +PSCI Implementation and Integration
> +-----------------------------------
> +vSYSTEM_SUSPEND
> +---------------
> +vSYSTEM_SUSPEND shall be implemented in
> +* do_psci_system_suspend() in arch/arm/vpsci.c
> +* Code independent from PSCI interface will be added in arch/arm/suspend.c
> +
> +The implementation shall include the following steps:
> +* Suspend the current (calling) vCPU. Consists of 2 major steps:
> +1) Reset context of vCPU and save entry point into PC and context ID into X0
> +(entry point and context ID are provided via vSYSTEM_SUSPEND arguments)
> +2) Block vCPU to ensure that it is not scheduled until it is unblocked by an
> +interrupt.
> +In step 1) above, the context is reset in order to prepare the vCPU for 
> resume,
> +i.e. to save vCPU context that matches reset values as expected by software 
> on
> +resume. This doesn't hold for PC and X0, since the PC contains resume entry
> +point and X0 contains context ID, as defined by PSCI.
> +* If the hardware domain made the call trigger Xen suspend, i.e.
> +  call machine_suspend() which will be implemented in arch/arm/suspend.c
> + (similar as the machine_restart() is implemented in arch/arm/shutdown.c)
> +
> +The function do_psci_system_suspend() shall be called from
> +* do_trap_psci() in arch/arm/traps.c
> +
> +CPU_OFF (physical CPUs)
> +-----------------------
> +The CPU_OFF function shall be implemented in
> +* call_psci_cpu_off() in arch/arm/psci.c
> +
> +The implementation shall consist just of making the SMC call to EL3.
> +
> +This function needs to be called when Xen generic code disables a non-boot 
> CPU.
> +When a CPU is disabled it will loop forever in while loop (stop_cpu() 
> function
> +which is already implemented in xen/arch/arm/smpboot.c). Call to
> +call_psci_cpu_off() shall be made before the CPU enters infinite loop.
> +
> +SYSTEM_SUSPEND (physical)
> +-------------------------
> +The SYSTEM_SUSPEND function shall be implemented in
> +* call_psci_system_suspend() in arch/arm/psci.c
> +
> +The implementation shall consist just of making the SMC call to EL3. The
> +entry_point_address argument of the SMC call needs to be an ARM architecture
> +resume address, which shall be implemented, e.g. as hyp_resume() in
> +arch/arm/arm64/entry.S. The call_psci_system_suspend() function does not 
> return.
> +On the resume, the execution flow continues from hyp_resume.
> +
> +The function needs to be called from machine_suspend() to finalize the 
> suspend
> +procedure.
> +
> +------------------
> +Additional Changes
> +------------------
> +
> +Suspend Flow
> +------------
> +The suspend procedure shall be implemented in
> +* machine_suspend() in arch/arm/suspend.c
> +
> +The implementation shall include the following steps:
> +* Move the execution to boot pCPU
> +* Set the system_state variable to SYS_STATE_suspend
> +* Disable watchdog
> +* Freeze domains by calling domain_pause() for each domain
> +* Disable non-boot CPUs by calling disable_nonboot_cpus()
> +* Disable interrupts
> +* Suspend timer
> +* Save GIC context. Shall be implemented in arch/arm/gic.c,
> +  include/asm-arm/gic.h and arch/arm/gic-v2.c (only GICv2 will be supported).
> +* Save CPU context. This shall be implemented in assembly, in hyp_suspend()
> +  in arch/arm/arm64/entry.S. The context consists of callee-saved general
> +  purpose registers, as well as few system registers. Context of registers 
> shall
> +  be saved in a statically allocated structure.
> +* Finalize the suspend by calling call_psci_system_suspend()
> +
> +Resume Flow
> +------------
> +The resume entry point shall be implemented in
> +* hyp_resume() in arch/arm/arm64/entry.S
> +The very beginning of the resume procedure has to be implemented in assembly.
> +It shall contain the following:
> +* Enable the MMU so that the structure containing CPU context which was 
> saved on
> +suspend can be accessed
> +* Restore CPU context (to match the values saved on suspend) and return into 
> C
> +* Set the system_state variable to SYS_STATE_resume
> +* Restore GIC context
> +* Resume timer
> +* Enable interrupts
> +* Enable non-boot CPUs by calling enable_nonboot_cpus()
> +* Thaw domains by calling domain_unpause() for each domain
> +* Enable watchdog
> +* Set the system_state variable to SYS_STATE_active
> +* Resume Dom0
> +
> +==========
> +References
> +==========
> +
> +[1] Power State Coordination Interface (ARM):
> +http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf
> -- 
> 2.13.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel