[Xen-devel] [PATCH RFC v2 0/8] Live migration for VMs with QEMU backed local storage

[Bruno Alvisio <bruno.alvisio@xxxxxxxxx>]

I am reviving this thread about the migration of VMs with local storage. I have 
worked on a solution to be able to migrate VMs that use QEMU as the backend 
disk driver. I have adapted the migration flow and piggybacked on the 
“drive-mirroring” capability already provided by QEMU.

Overview
1. The “xl migrate” command has an additional “-q” flag. When provided the 
local storage of the VM is mirrored to the destination during the migration 
process.
2. Internally, the modification consists on adding a new 
libxl__stream_read_state struct to the libxl__domain_create_state structure and 
libxl__stream_read_state structure to the libxl__domain_save_state struct.
3. Migration flow can now be divided into three phases:
   a. Phase One: Copies the necessary state to start a QEMU process on the 
destination. It is started with the “-incoming defer” option.
   b. Phase Two: Disk is mirrored using the QEMU embedded NBD server.
   c. Phase Three: Once the disk is completely mirrored, virtual RAM of the 
domain is live migrated to the destination. This phase most closely resembles 
to the current migration flow.
4. If the “-q” option is not provided the migration is equivalent to the 
current migration flow.

The new migration flow has follows the following major sequence of steps:
1. 1st stream copies the QEMU devices RAM from source to destination.
2. QEMU process is started on the destination with the option “-incoming 
defer”. (This creates the QEMU process but it doesn’t start running the main 
loop until “migrate incoming” command is executed)
3. “drive mirror” QMP command is executed so that the disk is mirrored to the 
destination node.
4. An event listener waits for the QMP BLOCK_JOB_READY event sent by QEMU which 
signals that the "disk mirror job" is complete.
5. 2nd Stream copies the virtual RAM from source to destination including QEMU 
state. At this point, the VM is suspended on source.
6. “migrate incoming” QMP command is executed on destination.
7. VM is restored in destination.

This is sample configuration file that I have used to test my branch:

name="tinycore"
disk=['/home/balvisio/tinycore.img,raw,xvda,w']
memory=128
builder='hvm'
vcpus=1
vfb = ['type=vnc']
vif= ['bridge=xenbr0']
boot='b'
acpi=1
device_model_version='qemu-xen'
serial='pty'
vnc=1
xen_platform_pci=0

Notes

1. Note that the configuration file uses "xen_platform_pci=0”. This is 
necessary so that the block device is seen by QEMU. Further modification should 
be made for the case "xen_platform_pci=1” if we still want to use NBD mirroring 
capability provided by QEMU.
2. The current branch has still many hardcoded values. Many of the can be 
easily removed:
        a. Port used for disk mirroring (11000)
    b. Name of the block devices. (ide0-hd0) Currently the branch only supports 
VM with on disk drive.
    c. Live migration memory transfer: The pages transferred by libxc is 
hardcoded. Only a VM with 128 MB of memory is supported.

Here is a link to the branch in Github: 
https://github.com/balvisio/xen/tree/feature/local_storage_migration

Any feedback/suggestion is appreciated.

Cheers,

Bruno

Signed-off-by: Bruno Alvisio <bruno.alvisio@xxxxxxxxx>

---
 tools/libxc/include/xenguest.h       |   6 +-
 tools/libxc/xc_nomigrate.c           |   6 +-
 tools/libxc/xc_sr_common.h           |   3 + 
 tools/libxc/xc_sr_restore.c          |  14 +-
 tools/libxc/xc_sr_save.c             | 117 +++++++++++++++-
 tools/libxc/xc_sr_save_x86_hvm.c     |   7 +-
 tools/libxc/xc_sr_stream_format.h    |   4 + 
 tools/libxl/libxl.h                  |  11 +-
 tools/libxl/libxl_create.c           | 189 +++++++++++++++++++++++--
 tools/libxl/libxl_dm.c               |  20 ++- 
 tools/libxl/libxl_dom_save.c         |  66 ++++++++-
 tools/libxl/libxl_domain.c           |   4 +-
 tools/libxl/libxl_internal.h         |  62 ++++++++-
 tools/libxl/libxl_qmp.c              | 261 +++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_save_callout.c     |  30 ++--
 tools/libxl/libxl_save_helper.c      |  13 +-
 tools/libxl/libxl_stream_read.c      |  17 ++- 
 tools/libxl/libxl_stream_write.c     |  28 +++-
 tools/libxl/libxl_types.idl          |   6 + 
 tools/ocaml/libs/xl/xenlight_stubs.c |   4 +-
 tools/xl/xl.h                        |   1 + 
 tools/xl/xl_cmdtable.c               |   3 +-
 tools/xl/xl_migrate.c                |  36 +++--
 tools/xl/xl_saverestore.c            |   2 +-
 tools/xl/xl_vmcontrol.c              |   5 +-
 25 files changed, 845 insertions(+), 70 deletions(-)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel