Re: [PATCH v14 0/3] VM forking

[Jan Beulich <jbeulich@xxxxxxxx>]

On 06.04.2020 17:20, Tamas K Lengyel wrote:
> The following series implements VM forking for Intel HVM guests to allow for
> the fast creation of identical VMs without the assosciated high startup costs
> of booting or restoring the VM from a savefile.
> 
> JIRA issue: https://xenproject.atlassian.net/browse/XEN-89
> 
> The fork operation is implemented as part of the "xl fork-vm" command:
>     xl fork-vm -C <config> -Q <qemu-save-file> -m <max-vcpus> <parent_domid>
>     
> By default a fully functional fork is created. The user is in charge however 
> to
> create the appropriate config file for the fork and to generate the QEMU save
> file before the fork-vm call is made. The config file needs to give the
> fork a new name at minimum but other settings may also require changes. 
> Certain
> settings in the config file of both the parent and the fork have to be set to
> default. Details are documented.
> 
> The interface also allows to split the forking into two steps:
>     xl fork-vm --launch-dm no \
>                -m <max-vcpus> \
>                -p <parent_domid>
>     xl fork-vm --launch-dm late \
>                -C <config_file_for_fork> \
>                -Q <qemu_save_file> \
>                <fork_domid>
> 
> The split creation model is useful when the VM needs to be created as fast as
> possible. The forked VM can be unpaused without the device model being 
> launched
> to be monitored and accessed via VMI. Note however that without its device
> model running (depending on what is executing in the VM) it is bound to
> misbehave or even crash when its trying to access devices that would be
> emulated by QEMU. We anticipate that for certain use-cases this would be an
> acceptable situation, in case for example when fuzzing is performed of code
> segments that don't access such devices.
> 
> Launching the device model requires the QEMU Xen savefile to be generated
> manually from the parent VM. This can be accomplished simply by connecting to
> its QMP socket and issuing the "xen-save-devices-state" command. For example
> using the standard tool socat these commands can be used to generate the file:
>     socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
>     { "execute": "qmp_capabilities" }
>     { "execute": "xen-save-devices-state", \
>         "arguments": { "filename": "/path/to/save/qemu_state", \
>                         "live": false} }
> 
> At runtime the forked VM starts running with an empty p2m which gets lazily
> populated when the VM generates EPT faults, similar to how altp2m views are
> populated. If the memory access is a read-only access, the p2m entry is
> populated with a memory shared entry with its parent. For write memory 
> accesses
> or in case memory sharing wasn't possible (for example in case a reference is
> held by a third party), a new page is allocated and the page contents are
> copied over from the parent VM. Forks can be further forked if needed, thus
> allowing for further memory savings.
> 
> A VM fork reset hypercall is also added that allows the fork to be reset to 
> the
> state it was just after a fork, also accessible via xl:
>     xl fork-vm --fork-reset -p <fork_domid>
> 
> This is an optimization for cases where the forks are very short-lived and run
> without a device model, so resetting saves some time compared to creating a
> brand new fork provided the fork has not aquired a lot of memory. If the fork
> has a lot of memory deduplicated it is likely going to be faster to create a
> new fork from scratch and asynchronously destroying the old one.
> 
> The series has been tested with Windows VMs and functions as expected. Linux
> VMs when forked from a running VM will have a frozen VNC screen. Linux VMs at
> this time can only be forked with a working device model when the parent VM 
> was
> restored from a snapshot using "xl restore -p". This is a known limitation.
> Also note that PVHVM/PVH Linux guests have not been tested. Forking most 
> likely
> works but PV devices and drivers would require additional wiring to set things
> up properly since the guests are unaware of the forking taking place, unlike
> the save/restore routine where the guest is made aware of the procedure.
> 
> Forking time has been measured to be 0.0007s, device model launch to be around
> 1s depending largely on the number of devices being emulated. Fork resets have
> been measured to be 0.0001s under the optimal circumstances.
> 
> New in v14:
>     minor adjustments
> 
> Patch 1 implements the VM fork
> Patch 2 implements fork reset operation
> Patch 3 adds the toolstack-side code implementing VM forking and reset
> 
> Tamas K Lengyel (3):
>   xen/mem_sharing: VM forking
>   x86/mem_sharing: reset a fork

I've applied these two, but ...

>   xen/tools: VM forking toolstack side

... since this one doesn't have any ack or alike I'll defer to
the tool stack maintainers here.

Jan