Bug 1640465

version of libvirt used?

We run into similar issue even after applying build that fixes bug 1635705 - this time only for the Hosted Engine VM (other VMs running on the RHV environment migrate without issues)
Could you take a look

(In reply to Sahina Bose from comment #7)
> We run into similar issue even after applying build that fixes bug 1635705 -
> this time only for the Hosted Engine VM (other VMs running on the RHV
> environment migrate without issues)
> Could you take a look

Is the fixed package installed on both source and destination? If so, can you please share output of "virsh domblklist --details" for the domain that is failing and also attach mount table from both the source and destination?

Please find the attached logs

Yes, this is a libvirt bug. While fixing bug 1632711 I did not realize that a dir in the path can be a symlink. Although, at this point it would be way easier if vdsm can just pass the VIR_MIGRATE_UNSAFE flag (which suppresses all checks like these).

Patch posted upstream:

https://www.redhat.com/archives/libvir-list/2018-October/msg00943.html

And I've just pushed the patch upstream:

commit c0790e3a09f57da0bd25c7eac4a35ed6e7e9e858
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Oct 18 14:57:19 2018 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 19 13:15:54 2018 +0200

    virfile: Take symlink into account in virFileIsSharedFixFUSE
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1640465
    
    Weirdly enough, there can be symlinks in the path we are trying
    to fix. If it is the case our clever algorithm that finds matches
    against mount table won't work. Canonicalize path at the
    beginning then.
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Erik Skultety <eskultet>

v4.8.0-117-gc0790e3a09

*** Bug 1644679 has been marked as a duplicate of this bug. ***

Created attachment 1505710 [details]
logs_migration_unsafe

I see the same migration error happens for regular VMs not HE environments build 4.3.

hypervisor-lynx18_vdsm.log:2018-10-11 18:47:26,188+0300 ERROR (migsrc/59b67fb0) [virt.vm] (vmId='59b67fb0-0d84-4c80-99a5-506d1bd263b2') migration destination error: Error creating the requested VM (migration:282)
hypervisor-lynx19_vdsm.log:2018-10-11 18:47:33,301+0300 ERROR (migsrc/a7703a0b) [virt.vm] (vmId='a7703a0b-c023-4576-a935-9efbd658cdf2') Unsafe migration: Migration without shared storage is unsafe (migration:282)
hypervisor-lynx19_vdsm.log:2018-10-11 18:47:33,881+0300 ERROR (migsrc/a7703a0b) [virt.vm] (vmId='a7703a0b-c023-4576-a935-9efbd658cdf2') Failed to migrate (migration:450)

attached logs logs_migration_unsafe.tar.gz

(In reply to Polina from comment #40)
> attached logs logs_migration_unsafe.tar.gz

libvirt logs are missing. Anyway, I suggest waiting for a build that contains fixes for this bug and re-runing your scenario.

Verified on 
libvirt-4.5.0-16.virtcov.el7.x86_64
qemu-kvm-rhev-2.12.0-27.el7.x86_64

1.Scenario 1
1. Mount glusterfs on dst and src host:
# mount -t glusterfs 10.66.4.119:/gvnew /mnt/mount

2. Make symlinks on dst and src hosts:
# ln -s /mnt/mount/glusterfs /var/lib/libvirt/images/glusterfs

3. Prepare a running VM whose image on the symlink dir:
# virsh domblklist jgao                                                                                                                        
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/glusterfs/A.qcow2

# virsh -k0 -K0 migrate jgao qemu+ssh://10.66.144.87/system --verbose                   
Migration: [100 %]

Migrate back:
# virsh -k0 -K0 migrate jgao qemu+ssh://10.73.196.199/system --verbose 
Migration: [100 %]

Check R/W on VM:
# echo xx>xx
# cat xx
xx


2.Scenario 2
1. Mount glusterfs on dst and src host:
# mount -t glusterfs 10.66.4.119:/gvnew /mnt/mount

2. Make symlinks on dst and src hosts:
# ln -s /mnt/mount/glusterfs /var/lib/libvirt/images/glusterfs

3. Create a backing chain of relative path:
# cd /var/lib/libvirt/images

# qemu-img create gls.copy -b glusterfs/A.qcow2 -f qcow2 -o backing_fmt=qcow2
Formatting 'gls.copy', fmt=qcow2 size=10737418240 backing_file=glusterfs/A.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

Set VM image as top layer of backing chain
# virsh domblklist jgao                                                               
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/gls.copy


4.Start VM and migration with --copy-storage-inc
# virsh start jgao                                                               
Domain jgao started

# virsh -k0 -K0 migrate jgao qemu+ssh://10.66.144.87/system --verbose --copy-storage-inc
Migration: [100 %]

Migrate back:
# virsh -k0 -K0 migrate jgao qemu+ssh://10.73.196.199/system --verbose --copy-storage-inc
Migration: [100 %]


Works as expected

Created attachment 1568313 [details]
code coverage , missed 1 line

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2294