Bug 1635705

To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2018-October/msg00071.html

Test on libvirt-4.5.0-10.virtcov.el7_6.1.x86_64 qemu-kvm-rhev-2.12.0-18.el7.x86_64.
The migration on glusterfs.fuse will fail if glusterfs.fuse is not the last row on mount table. 
For example, this mount table will cause migration failure:
10.66.4.183:/gv0 /mnt fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
root.4.183:/tmp/mkdir /tmp/br0 fuse.sshfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

While this will be successful:
root.4.183:/tmp/mkdir /tmp/br0 fuse.sshfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
10.66.4.183:/gv0 /mnt fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0



My mount table:
10.66.4.183:/gv0 /mnt fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
root.4.183:/tmp/mkdir /tmp/br0 fuse.sshfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

I checked the function virFileIsSharedFixFUSE and find the strings in char **mounts will be changed after mb is changed by getmntent_r(f, &mb, mntbuf, sizeof(mntbuf):

When mb.mnt_type == "fuse.glusterfs" and then the code jump to: 
    if (VIR_APPEND_ELEMENT_COPY(mounts, nmounts, mb.mnt_dir) < 0)
After this, the *mounts value is 
(gdb) p *mounts                                                                       
$38 = 0x7f51fa2241d1 "/mnt"

Then after the next while (getmntent_r(f, &mb, mntbuf, sizeof(mntbuf))), the *mount value becomes:
(gdb) p *mounts                                                                       
$43 = 0x7f51fa2241d1 "/tmp/mkdir"

The same as mnt_dir of the last row in mount table.

So I guess the reason is VIR_APPEND_ELEMENT_COPY only stores the address of a item not the item's value.

A patch to fix commet6:
https://www.redhat.com/archives/libvir-list/2018-October/msg00511.html

Indeed, I've proposed an alternative solution:

https://www.redhat.com/archives/libvir-list/2018-October/msg00554.html

Created attachment 1492317 [details]
Full mount table of comment6

To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2018-October/msg00369.html

SC1: Migration an VM based on mounted glusterfs and then migrate back. The src image dir is nest mounted glsuterfs while the glusterfs mount point of dst is not the last one in mount table.

1. Prepare mount points
1.1 Mount points on src host
# sshfs -o allow_other,nonempty root.4.183:/tmp/mkdir /mnt
# mkdir /mnt/glusterfs
# mount -t glusterfs 10.66.4.183:/gv0 /mnt/glusterfs

Check mount table:
# cat /proc/mounts|tail
hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
/dev/mapper/mpathb1 /boot xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
tmpfs /run/user/42 tmpfs rw,seclabel,nosuid,nodev,relatime,size=788556k,mode=700,uid=42,gid=42 0 0
tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=788556k,mode=700 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
root.4.183:/tmp/mkdir /mnt fuse.sshfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
10.66.4.183:/gv0 /mnt/glusterfs fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0



1.2 Mount points on dst host
# mkdir /mnt/glusterfs
# mount -t glusterfs 10.66.4.183:/gv0 /mnt/glusterfs
# mkdir /tmp/ssh
# sshfs -o allow_other,nonempty root.4.183:/tmp/mkdir /tmp/ssh/
root.4.183's password: 

Check mount table:
# cat /proc/mounts |tail
mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
/dev/sda1 /boot xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
/dev/mapper/rhel-home /home xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=795072k,mode=700 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
10.66.4.183:/gv0 /mnt/glusterfs fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
root.4.183:/tmp/mkdir /tmp/ssh fuse.sshfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

2. migrate and migrate back
# virsh -k0 -K0 migrate rhel7 qemu+ssh://root@fjin-5-190/system --verbose
Migration: [100 %]

# virsh -k0 -K0 migrate rhel7 qemu+ssh://root.me/system --verbose
root.me's password: 
Migration: [100 %]

Check R/W in VM:
(VM) # echo xx>xx
(VM) # cat xx
xx


SC2 Create external snapshot and check the permission
# virsh snapshot-create-as rhel7 s1 --disk-only 
Domain snapshot s1 created

# ls /mnt/glusterfs/A.s1 -al
-rw-------. 1 qemu qemu 524288 Oct 16 09:59 /mnt/glusterfs/A.s1

Check R/W in VM:
(VM) # echo xx>xx
(VM) # cat xx
xx

SC3: Save and restore on the gluster mount point:
# virsh -k0 -K0 save rhel7 /mnt/glusterfs/new.sav                        
Domain rhel7 saved to /mnt/glusterfs/new.sav

# virsh restore /mnt/glusterfs/new
viDomain restored from /mnt/glusterfs/new.sav

Check R/W in VM:
(VM) # echo xx>xx
(VM) # cat xx
xx

More scenarios:
Assume that /mnt/glusterfs is the shared glusterfs mount point.
SC1: Shallow copy storage to gluster mount point:
# qemu-img create -b /mnt/glusterfs/A.qcow2 -o backing_fmt=qcow2 /mnt/glusterfs/shallow -f qcow2
Formatting '/mnt/glusterfs/shallow', fmt=qcow2 size=10737418240 backing_file=/mnt/glusterfs/A.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

# virsh blockcopy rhel7 vda /mnt/glusterfs/shallow --reuse-external --shallow --pivot --transient-job --verbose --wait 
Block Copy: [100 %]
Successfully pivoted

# ls -alZ /mnt/glusterfs/shallow
-rw-r--r--. qemu qemu system_u:object_r:fusefs_t:s0    /mnt/glusterfs/shallow

Check R/W in VM:
(VM)# echo xx>xx
(VM)# cat xx
xx


SC2: Incremental storage migration with base image on gluster shared mount points
Prepare a backing chain storage file, whose top layer is non-shared file and basic layer is shared file.

# qemu-img info /tmp/A.s1 
image: /tmp/A.s1
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 81M
cluster_size: 65536
backing file: /mnt/glusterfs/A.qcow2
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Create backing file on dst host:
(dst host)# qemu-img create -b /mnt/glusterfs/A.qcow2 -o backing_fmt=qcow2 /tmp/A.s1 -f qcow2
Formatting '/tmp/A.s1', fmt=qcow2 size=10737418240 backing_file=/mnt/glusterfs/A.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
 
Start VM on src with the top layer image:
# virsh start rhel7 
Domain rhel7 started

# virsh domblklist rhel7
Target     Source
------------------------------------------------
vda        /tmp/A.s1

Do migration with --copy-storage-inc
# virsh -k0 -K0 migrate rhel7 qemu+ssh://root@fjin-5-190/system --verbose --copy-storage-inc
Migration: [100 %]

Check R/W in VM:
(VM)# echo xx>xx
(VM)# cat xx
xx

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3339