[Xen-devel] [RFC V10 2/4] domain snapshot overview

[Chunyan Liu <cyliu@xxxxxxxx>]

Changes to V9:
  * add description in details about domain snapshot and disk snapshot:
    possible user cases, background knowledge about disk snapshots
    implementation, domain snapshot cases, libvirt toolstack support
    status, xl support TODO, etc.

=====================================================================

1. Introduction

There are several types of snapshots:

 disk snapshot
     Contents of disks are saved at a given point of time, and can be
     restored back to that state.

     On a running guest, a disk snapshot is likely to be only
     crash-consistent rather than clean (that is, it represents the
     state of the disk on a sudden power outage, and may need fsck or
     journal replays to be made consistent).

     On a paused guest, with mechanism of quiesing disks (that is,
     all cached data written to disk), a disk snapshot is clean.

     On an inactive guest, a disk snapshot is clean if the disks were
     clean when the guest was last shut down.

     Disk snapshots exist in two forms: internal (file formats such as
     qcow2 track both the snapshot and changes since the snapshot in a
     single file) and external (the snapshot is one file, and the
     changes since the snapshot are in another file).

 memory state (or VM state)
     Tracks only the state of RAM and all other resources in use by
     the VM. If the disks are unmodified between the time a VM state
     snapshot is taken and restored, then the guest will resume in a
     consistent state; but if the disks are modified externally in
     the meantime, this is likely to lead to data corruption.

 system checkpoint (domain snapshot)
     A combination of disk snapshots for all disks as well as VM
     memory state, which can be used to resume the guest from where
     it left off with symptoms similar to hibernation (that is, TCP
     connections in the guest may have timed out, but no files or
     processes are lost).

     A system checkpoint can contain disk snapshots + VM state; or
     contains disk snapshots only without VM state, in this case,
     it should quiesce all disks before taking disk snapshots. The
     latter case is also referred as 'disk-only domain snapshot'.

VM state (memory) snapshots are created by 'domain save', and restore
via 'domain restore'.

Disk snapshot can be created by many external tools, like qemu-img,
vhd-util and lvm, etc.

Domain snapshot (including disk-only domain snapshot) will be handled
by 'domain snapshot' functionality.

Domain snapshot with memory state (as VM state) includes live and
non-live mode according to the VM downtime difference. Live mode will
try best to reduce downtime of the guest, but as a result will increase
size of the memory dump file.


2. Domain Snapshot User Cases

Domain snapshot can be used in following cases:

* Domain snapshot can be used as a domain backup. It can preserve the
  VM status at a certain point and able to roll back to it.

* Domain snapshot can support 'gold image' type deployments, i.e.
  where you create one baseline single disk image and then clone it
  multiple times to deploy lots of guests; when you create a domain
  snapshot, with it as gold domain snapshot (duplicate multiple times),
  one can restore from the gold domain snapshot mulitple times for
  different reasons.

* Disk-only domain snapshot can be used as backup out of domain,
  i.e. taking a disk-only domain snapshot and then run you usual backup
  software on the disk snapshots (which is now unchanging, which
  is handy); one can backup that static version of the disk out of band
  from the domain itself (e.g. can attach it to a separate backup VM).

3. Domain Snapshot Operations

Generally, domain snapshot includes 4 kinds of operations:

* create a domain snapshot

   create domain snapshot under different conditions:
   - domain is live, save vm state (live), disk snapshot
   - domain is live, save vm state (non-live), disk snapshot
   - domain is live, disk-only snapshot (need quiecing disks)
   - domain is offline, disk-only snapshot
   (under each above condition, disk snapshot can be
    internal/external.)

* revert (roll back to) a domain snapshot

   revert domain snapshot under different conditions:
   - domain is live, has vm state, all internal disk snapshots
   - domain is live, has vm state, has external disk snapshots
   - domain is live, no vm state, all internal disk snapshots
   - domain is live, no vm state, has external disk snapshots
   - domain is offline, has vm state, all internal disk snapshots
   - domain is offline, has vm state, has external disk snapshots
   - domain is offline, no vm state, all internal disk snapshots
   - domain is offline, no vm state, has external disk snapshots

* delete a domain snapshot

   delete domain snapshot under following conditions:
   - domain is live, not in a snapshot chain
   - domain is live, in a snapshot chain
   - domain is offline, not in a snapshot chain
   - domain is offline, in a snapshot chain

* list domain snapshot(s)
   list domain snapshot(s) contains:
   - list a single domain snapshot
   - list all domain snapshots
   - list snapshot(s) in details

Since domain snapshot includes disk snapshot, expand disk snapshot
operations here for later understanding.

# Disk Snapshot Operations

 * Create disk snapshot
 * Delete disk snapshot
 * Revert (apply) disk snapshot
 * List disk snapshots

 Tools:
   - Internal disk snapshot
     - 'qcow2' format
       * domain is live, through qmp command
       * domain is offline, qemu-img
     - other formats, not supported
   - External disk snapshot
     - 'lvm' format, lvm tool
     - 'vhd' format, vhd-util or qmp/qemu-img command
     - other formats
       * domain is live, qmp command
       * domain is offline, qemu-img command

# Existing internal/external disk snapshot implementation

  Add a few words to explain the implementation of internal
  and external disk snapshots, to help understanding.

   * Internal disk snapshot:

     Supported by 'qcow2' format only. The implementation is in qemu.
     According to qemu code, creating a snapshot will add a copy of
     cluster table, and increase refcounts of the clusters, e.g. if
     refcount > 1, will COW on a write operation; when deleting data,
     it will decrease refcount, not delete data until refcount = 0.
     [1]

     So, deleting/reverting an internal snapshot (even in a snapshot
     chain) is very straight forward, won't affect other snapshots.
     [2]

   * External disk snapshot:

     'lvm' format:
        implemented by lvm tool, using backing file mechanism.
        After snapshot command, original disk image becomes
        backing file and the new image will be created to track
        the new changes.

        If it is used in a domain, after the snapshot, domain
        should use the new image to replace the original image
        for further operations.

        e.g.:
        Before snapshot:
        domain --> disk A

        take disk snapshot:
        disk A (Base) <- disk B (Active)

        after snapshot:
        domain --> disk B

        Lvm tool doesn't support snapshot of snapshot.

     'vhd' format:
        implemented by 'vhd-util', qemu qmp command can also do
        the work. As lvm, both vhd-util and qemu qmp command
        use backing file mechanism too.

     Other formats:
        supported by qemu qmp command. Same as above,
        using backing file mechanism too.


     In summary, all external disk snapshot tools are implemented
     with backing file mechanism.

     snapshot of snapshot: becomes a backing file chain, like:

     RootBase <-sn1 <- sn2 <- sn3  <- new image A (active)
                 |
                  <--sn4 <- new image B (active)

     Compared with internal disk snapshot, deleting/reverting an
     external disk snapshot is more tricky.

     Deleting involves complicated process dealing with additional
     files, merge 'base' to 'top' (or say, 'parent' to 'children').
     Qemu supplies method to do that, refer to qmp commands:
     block-stream, block-commit, or qemu-img rebase, qemu-img commit.

     example:
     RootBase <- sn1 <- sn2 <- Active

     To delete sn1:
     [online]
     # use QMP 'block-stream' to populate data of sn1 to sn2
     # rm sn1

     [offline]
     # qemu-img rebase -b RootBase sn2 (sn1 data will be merged to sn2)
     # rm sn1

     Reverting involves duplicate/copy backing file, since backing
     file is read-only and could not be changed.

     Might because of the complexity, currently libvirt qemu driver
     doesn't support deleting or reverting to a domain snapshot
     containg external disk snapshot, but it supplies commands like:
     'virsh block-pull', 'virsh block-commit', 'virsh block-copy' to
     help users dealing with external snapshots.


4. Libvirt toolstack support status

Now, libvirt qemu driver supports domain snapshot functionality, but
not all cases. Following is the support status:

[support: Y; not support: N]

* create a domain snapshot

    [Y] - domain is live, save vm state (live), disk snapshot
    [Y] - domain is live, save vm state (non-live), disk snapshot
    [Y] - domain is live, disk-only snapshot
    [Y] - domain is offline, disk-only snapshot
          (under each above condition, disk snapshot can be
          internal/external.)

    libvirt toolstack maintains snapshot info and the chain
    relationship.

* revert (roll back to) a domain snapshot

    [Y] - domain is live/offline, has/no vm state, all internal disk snapshots
    [N] - domain is live/offline, has/no vm state, has external disk snapshots

* delete a domain snapshot

    [Y] - domain is live/offline, not in a snapshot chain,
          all internal disk snapshots
    [N] - domain is live/offline, not in a snapshot chain,
          has external disk snapshots
    [Y] - domain is live/offline, in a snapshot chain,
          no one in the chain contains external disk snapshot.
          delete this snapshot or together with children snapshots.
    [N] - domain is live/offline, in a snapshot chain,
          someone in the chain contains external disk snapshot.

* list domain snapshot(s)
    [Y] - list a single domain snapshot
    [Y] - list all domain snapshots
    [Y] - list snapshot(s) in details


5. xl toolstack planning support

Following the existing xl idioms of managing storage and saved
VM images via existing CLI command (qemu-img, lvcreate, ls, mv,
cp etc), xl snapshot functionality would be kept as simple as
possible.

xl will support creating snapshot and reverting to a snapshot.

But it won't manage snapshots information, as xl doesn't maintain
saved images created by 'xl save'. xl will have no idea of
the existence of domain snapshots and the chain relationship
among snapshots, so it depends on user to take care of the
snapshots and the snapshot chain relationship, and delete
snapshots.

xl won't support deleting snapshot and listing snapshots.

And talking about disk-only snapshot, since xl only concerns
active (live) domains, even when domain is paused, no mechanism
to quiesce disks, that means, taking a disk-only snapshot and
then resume, it is as if the guest had crashed. For this reason,
currently xl taking disk-only snapshot is useless.

Following will be the planning support:

[support: Y; not support: N]

* create a domain snapshot

    [Y] - domain is live, save vm state (live), disk snapshot
    [Y] - domain is live, save vm state (non-live), disk snapshot
    [N] - domain is live, disk-only snapshot
          (under each above condition, disk snapshot can be
          internal/external.)

* revert (roll back to) a domain snapshot

    [Y] - domain is live, has/no vm state, all internal disk snapshots
    [Y?] - domain is live, has/no vm state, has external disk snapshots

* delete a domain snapshot
    [N]

* list domain snapshots
    [N]


# Interaction with other operations:

  Don't support creating domain snapshot when domain is shutdowning
  or dying.


6. xl toolstack planning implementation outline

# General Requirements:

 * ability to save/restore domain memory
 * ability to create/delete/revert disk snapshot
 * ability to parse user config file

# xl toolstack general workflow

 Create a snapshot:
 - parse user cfg file if passed in
 - check snapshot operation is allowed or not
 - save domain, saving memory status to file (refer to: save_domain)
   [if quiecing disks mechanism implemented in future, then can
    support disk-only snapshot, in that case, instead of saving
    memory, it will be:
    - pause domain, quiecing all disks]
 - take disk snapshot (e.g. call qmp command)
 - unpause domain

 Revert to snapshot:
 - parse use cfg file (xl doesn't manage snapshots, so it has no
   idea of snapshot existence. User MUST supply configuration file)
 - destroy this domain
 - create a new domain from snapshot info
   - apply disk snapshot (e.g. call qemu-img)
   - a process like restore domain

[1] https://people.gnome.org/~markmc/qcow-image-format.html
[2] https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel