Bug 1210352

*** Bug 1204040 has been marked as a duplicate of this bug. ***

*** Bug 1232919 has been marked as a duplicate of this bug. ***

I've just pushed the patches upstream:

commit a4e92f9e14d0a81ceaa2bc8d0c423522fb455184
Author:     Pavel Boldin <pboldin>
AuthorDate: Tue Jun 16 01:42:11 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    virsh: selective block device migration
    
    Add `virsh migrate' option `--migrate-disks' that allows CLI user to
    explicitly specify block devices to migrate.
    
    Signed-off-by: Pavel Boldin <pboldin>
    Signed-off-by: Michal Privoznik <mprivozn>

commit 93a19e283e5a147d147d84383aaaa8be63b6a68d
Author:     Pavel Boldin <pboldin>
AuthorDate: Tue Jun 16 01:42:10 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    qemu: migration: selective block device migration
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1203032
    
    Implement a `migrate_disks' parameters for the QEMU driver. This multi-
    value parameter can be used to explicitly specify what block devices
    are to be migrated using the NBD server. Tunnelled migration using NBD
    is to be done.
    
    Signed-off-by: Pavel Boldin <pboldin>
    Signed-off-by: Michal Privoznik <mprivozn>

commit 5eb03b6ea07d6e9178f5b8510681ea9ec9ca422f
Author:     Pavel Boldin <pboldin>
AuthorDate: Tue Jun 16 01:42:09 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    util: add virTypedParamsAddStringList
    
    The `virTypedParamsAddStringList' function provides interface to add a
    NULL-terminated array of string values as a multi-value to the params.
    
    Signed-off-by: Pavel Boldin <pboldin>
    Signed-off-by: Michal Privoznik <mprivozn>

commit 952907f5401512013a1296d7888b1217abecff76
Author:     Pavel Boldin <pboldin>
AuthorDate: Tue Jun 16 01:42:08 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    util: virTypedParams{Filter,GetStringList}
    
    Add multikey API:
    
     * virTypedParamsFilter that filters all the parameters with specified name.
     * virTypedParamsGetStringList that returns a list with all the values for
       specified name and string type.
    
    Signed-off-by: Pavel Boldin <pboldin>
    Signed-off-by: Michal Privoznik <mprivozn>

commit e9ef8565205bdb1438c9a5e40a3d54e29f562b4e
Author:     Pavel Boldin <pboldin>
AuthorDate: Tue Jun 16 01:42:07 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    util: multi-value parameters in virTypedParamsAdd*
    
    Allow multi-value parameters to be build using virTypedParamsAdd*
    functions by removing check for duplicates.
    
    Signed-off-by: Pavel Boldin <pboldin>
    Signed-off-by: Michal Privoznik <mprivozn>

commit a5250449de8d570992744f309fefaaaa6f8212cd
Author:     Pavel Boldin <pboldin>
AuthorDate: Tue Jun 16 01:42:06 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    util: multi-value virTypedParameter
    
    The `virTypedParamsValidate' function now can be instructed to allow
    multiple entries for some of the keys. For this flag the type with
    the `VIR_TYPED_PARAM_MULTIPLE' flag.
    
    Add unit tests for this new behaviour.
    
    Signed-off-by: Pavel Boldin <pboldin>
    Signed-off-by: Michal Privoznik <mprivozn>

commit cb7297c150639e9f70e414f3a82d1cde9fa3d9d6
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Jun 16 01:42:05 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    qemuMigrationDriveMirror: Force raw format for NBD
    
    When playing with disk migration lately, I've noticed this warning in
    domain logs:
    
    WARNING: Image format was not specified for 'nbd://masina:49153/drive-virtio-disk0' and probing guessed raw.
             Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
             Specify the 'raw' format explicitly to remove the restrictions.
    
    So I started digging into qemu source code to see what has triggered
    the warning. I'd expect qemu to know formats of guest's disks since we
    tell them on command line. This lead me to qmp_drive_mirror() where
    the following can be found:
    
        if (!has_format) {
            format = mode == NEW_IMAGE_MODE_EXISTING ? NULL : bs->drv->format_name;
        }
    
    So, format is automatically initialized from the disk iff mode !=
    "existing". Unfortunately, in migration we are tied to use this mode
    (NBD doesn't support creating new images). Therefore the only way to
    avoid this warning is to pass format. The discussion on the mail-list [1]
    resulted in the code that always forces NBD export as "raw" format.
    
    [1] https://www.redhat.com/archives/libvir-list/2015-June/msg00153.html
    Signed-off-by: Michal Privoznik <mprivozn>
    Signed-off-by: Pavel Boldin <pboldin>

commit 9c5efd1afd49a57c4d17fc9245995fc4e1e6f653
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Jun 16 01:42:04 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    qemuMigrationBeginPhase: Fix function header indentation
    
    This function is returning a string (domain XML). Since d3ce7363
    when it was first introduced, it was indented incorrectly:
    
    static char
    *qemuMigrationBeginPhase(..)
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 1049a8d8b4fa3efb63cfd39ae924f5ada3338115
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Jun 16 01:42:03 2015 +0300
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 16:46:09 2015 +0200

    virDomainDiskGetSource: Mark passed disk as 'const'
    
    The disk is not changed anywhere in the function. Mark this fact
    in the function header too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

Test this with build:
libvirt-1.2.17-7.el7.x86_64
qemu-kvm-rhev-2.3.0-22.el7.x86_64

scenario 1(passed) : only copy not-shared images
step:
1: prepare a guest with mix of shared and no-shared storage.
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/kvm-rhel7.1-x86_64-qcow2.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/1.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/migrate/2.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>

vda and vdb are no-shared storage. vdc is shareable.
2: start guest and do migration
# virsh migrate rhel7 --live qemu+ssh://$target_ip/system --migrate-disks vda,vdb --verbose --copy-storage-all
Migration: [100 %]

check libvirtd log:
2015-09-07 09:20:31.405+0000: 25164: debug : virJSONValueToString:1795 : result={"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk0","writable":true},"id":"libvirt-105"}
.....
2015-09-07 09:20:31.412+0000: 25164: debug : virJSONValueToString:1795 : result={"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk1","writable":true},"id":"libvirt-106"}

nbd only add disk0 and disk1.

scenario 2(passed): not add migrate-disks
step: 
1: same guest with scenario 1

2: start guest and do migration without use --migrate-disks
# virsh migrate rhel7 --live qemu+ssh://$target_ip/system --verbose --copy-storage-all
Migration: [100 %]
3: check target guest, worked well.
4: check libvirtd log
2015-09-07 09:30:55.249+0000: 25161: debug : virJSONValueToString:1795 : result={"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk0","writable":true},"id":"libvirt-105"}
...
2015-09-07 09:30:55.258+0000: 25161: debug : virJSONValueToString:1795 : result={"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk1","writable":true},"id":"libvirt-106"}
...
2015-09-07 09:30:55.267+0000: 25161: debug : virJSONValueToString:1795 : result={"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk2","writable":true},"id":"libvirt-107"}
...
ndb add 3 disks.

scenario 3(passed): add one not-shared disk in migrate-disks
step:
1: same guest with scenario 1
2: start guest and do migration
# virsh migrate rhel7 --live qemu+ssh://$target_ip/system --migrate-disks vda --verbose --copy-storage-all
Migration: [100 %]
3: check libvirtd log
2015-09-07 09:55:55.944+0000: 25160: debug : virJSONValueToString:1795 : result={"execute":"nbd-server-add","arguments":{"device":"drive-virtio-disk0","writable":true},"id":"libvirt-105"}

only add 1 disk.

Hi Michal:
Do you think these scenarios are enough to verifiy this bug? and i have two questions about this:
1:  i migrate many times, sometimes i met 
error: Cannot access storage file '/var/lib/libvirt/images/1.img' (as uid:107, gid:107): No such file or directory, 
after i pool-refresh the pool of target machine, migration worked well(because of target pool have some vols exists,but image is removed), is it worked as design? how about auto refresh target pool before migration? 
2: If not specify disk in migrate-disks, what way libvirtd used to copy not-shared images to target machine?

(In reply to zhe peng from comment #8)
> <snip/>
> 
> Hi Michal:
> Do you think these scenarios are enough to verifiy this bug?

Yes

> and i have two questions about this:
> 1:  i migrate many times, sometimes i met 
> error: Cannot access storage file '/var/lib/libvirt/images/1.img' (as
> uid:107, gid:107): No such file or directory, 
> after i pool-refresh the pool of target machine, migration worked
> well(because of target pool have some vols exists,but image is removed), is
> it worked as design? how about auto refresh target pool before migration?

Yes. I mean, storage pre-creation is done via our storage driver. Therefore if users wants to use the feature they should have a storage pools defined over the disks targets. The reason why you are seeing this is because you removed the file but not volume in libvirt. I don't think that autorefresh is a good idea - it's a policy decision whether mgmt application can postpone the migration for a couple of minutes (worst case scenario) while libvirt refreshes a storage pool (the pool can be on distant host on the network, or something).

> 2: If not specify disk in migrate-disks, what way libvirtd used to copy
> not-shared images to target machine?

The old way, that is everything bug shared and read-only disks are migrated.

Very thanks Michal's help, per comment 8 and comment 9, move to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html