Bug 869944 - Storage Live Migration with Live Guest Migration in libvirt
Storage Live Migration with Live Guest Migration in libvirt
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Michal Privoznik
Virtualization Bugs
: FutureFeature
: 845676 (view as bug list)
Depends On: 845675
Blocks: 845674 845679 845676 927252 1056726 1113727 1122703
  Show dependency treegraph
 
Reported: 2012-10-25 03:58 EDT by Paolo Bonzini
Modified: 2014-07-23 15:57 EDT (History)
15 users (show)

See Also:
Fixed In Version: libvirt-1.0.4-1.el7
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 845675
: 927252 (view as bug list)
Environment:
Last Closed: 2014-06-13 09:20:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Paolo Bonzini 2012-10-25 03:58:54 EDT
Support in libvirt to move a guest's storage and do live migration of the guest at the same time, using QEMU's support for disk mirroring and an NBD server running on the destination.

The process should be as follows:

1) User invokes libvirt's migrate functionality.

2) libvirt checks that no block jobs are active on the source.

3) libvirt starts the destination QEMU and sets up the NBD server using the nbd-server-start and nbd-server-add commands.

4) libvirt starts drive-mirror with a destination pointing to the remote NBD server, for example nbd:host:port:exportname=diskname (where diskname is the -drive id specified on the destination).

5) once all mirroring jobs reach steady state, libvirt invokes the migrate command.

6) once migration completed, libvirt invokes the nbd-server-stop command on the destination QEMU.
Comment 1 Dave Allan 2012-11-13 11:55:03 EST
Paolo, do you have qemu code we can start integrating with, either in your tree or committed upstream?
Comment 2 Paolo Bonzini 2012-11-14 05:42:55 EST
Yes, base functionality is now upstream.  Only performance improvements are missing.
Comment 3 Paolo Bonzini 2012-11-15 10:48:29 EST
*** Bug 845676 has been marked as a duplicate of this bug. ***
Comment 4 Michal Privoznik 2012-11-27 13:51:31 EST
Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2012-November/msg01196.html
Comment 5 Michal Privoznik 2013-02-06 09:20:01 EST
Another version:

https://www.redhat.com/archives/libvir-list/2013-February/msg00277.html
Comment 6 Michal Privoznik 2013-03-07 06:00:17 EST
Patches that replace current implementation (block copying through 'migrate' monitor command arguments in favour of 'nbd-*' monitor commands) are pushed upstream now:

commit 1e54685fc7c58e1db677bda18dd03d4499458535
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Wed Jan 30 17:53:37 2013 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 08:51:30 2013 +0100

    qemu_migration: Cancel running jobs on failed migration
    
    If a migration fails, we need to stop all block jobs running so
    qemu doesn't try to send data to destination over and over again.

commit ae21b9bde690d572cb9ca97e4b5c446eef83d3a6
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Tue Nov 27 16:34:24 2012 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 08:42:57 2013 +0100

    qemu_migration: Stop NBD server at Finish phase
    
    At the end of migration, it is important to stop NBD
    server and thus release all allocated resources.

commit 7b7600b3e6734117b3db39ba5c31ae33a40fb5bb
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Fri Nov 23 15:42:51 2012 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 08:42:54 2013 +0100

    qemu_migration: Introduce qemuMigrationDriveMirror
    
    This function does the source part of NBD magic. It
    invokes drive-mirror on each non shared and RW disk with
    a source and wait till the mirroring process completes.
    When it does we can proceed with migration.
    
    Currently, an active waiting is done: every 500ms libvirt
    asks qemu if block-job is finished or not.  However, once
    the job finishes, qemu doesn't report its progress so we
    can only assume if the job finished successfully or not.
    The better solution would be to listen to the event which
    is sent as soon as the job finishes. The event does
    contain the result of job.

commit 86d90b3abd0afcadd60c76020d50665a07b9a0df
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Jan 31 14:48:06 2013 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 08:25:09 2013 +0100

    qemu_migration: Introduce qemuMigrationStartNBDServer()
    
    We need to start NBD server and feed it with all non-<shared/>,
    RW and source-full disks. Moreover, with new virPortAllocator we
    must ensure the borrowed port for NBD server will be returned if
    either migration completes or qemu process is torn down.

commit f1748e34e2ed86b7ce622921dfe74621eba38f87
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Jan 31 14:47:49 2013 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 08:16:42 2013 +0100

    qemu: Introduce nbd-server-stop command
    
    This will be used after all migration work is done
    to stop NBD server running on destination.  It
    doesn't take any arguments, just issues a command.

commit c833d8111d56cdd3d46c49ebcb5ffc371dd548b5
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Nov 22 16:17:13 2012 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 08:06:37 2013 +0100

    qemu: Introduce nbd-server-add command
    
    This will be used with new migration scheme.
    This patch creates basically just monitor stub
    functions. Wiring them into something useful
    is done in later patches.

commit bb6359e8d48529d60a7194c13bd25de64412dba9
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Nov 22 16:08:52 2012 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 07:58:13 2013 +0100

    qemu: Introduce nbd-server-start command
    
    This will be used with new migration scheme.
    This patch creates basically just monitor stub
    functions. Wiring them into something useful
    is done in later patches.

commit 121d4cfb9a57289154a7290fa786d28e13eab9de
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Tue Jan 29 13:38:50 2013 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 07:49:56 2013 +0100

    Introduce NBD migration cookie
    
    This migration cookie is meant for two purposes. The first is to be sent
    in begin phase from source to destination to let it know we support new
    implementation of VIR_MIGRATE_NON_SHARED_{DISK,INC} so destination can
    start NBD server. Then, the second purpose is, destination can let us
    know, on which port the NBD server is running.

commit e9a6704f999be69d3a98108fe1e67fef8e794e54
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Wed Nov 21 14:28:44 2012 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Sat Feb 23 07:33:43 2013 +0100

    qemu: Introduce NBD_SERVER capability
    
    This just keeps track whether qemu knows nbd-server-* commands
    so we can use it during migration or not.

v1.0.2-263-g1e54685


However, there is still this limitation left - users are required to pre-create storage themselves. I am working on dropping this limitation. Hence I don't think this one should be moved to POST yet.
Comment 8 Dave Allan 2013-03-25 08:49:04 EDT
> However, there is still this limitation left - users are required to
> pre-create storage themselves. I am working on dropping this limitation.
> Hence I don't think this one should be moved to POST yet.

Please open a separate RFE for storage pre-creation.
Comment 11 zhe peng 2013-10-17 04:00:28 EDT
thanks Michal's help

I verify this with below :
build:
 libvirt-1.1.1-9.el7.x86_64
 kernel-3.10.0-33.el7.x86_64
 qemu-kvm-1.5.3-9.el7.x86_64

step:
1: prepare two host for migration,source and target.
2: prepare a guest on source, pre-creation storage with same name file on target.
3: run cmd:
1) full disk copy
# virsh migrate --live rhel6 qemu+ssh://10.66.106.25/system --verbose --copy-storage-all
Migration: [100 %]

check libvirtd log on target host:
....
2013-10-17 06:29:48.185+0000: 413: debug : qemuMonitorNBDServerStart:3681 : mon=0x7f3e5c004eb0 host=:: port=5901
....
2013-10-17 06:29:48.187+0000: 413: debug : qemuMonitorNBDServerAdd:3703 : mon=0x7f3e5c004eb0 deviceID=drive-ide0-0-0

2) incremental copy
same with full disk copy.

the verification passed.
But i found sometime migration will failed when spice client connected,i will open another bug to track. this bug move to verified.
Comment 14 Ludek Smid 2014-06-13 09:20:47 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.