Bug 1740506

Please upload libvirtd log with following log filters:
log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util 1:security"

Created attachment 1603251 [details]
libvirtd.log

Libvirt log from start guest to migrate.

Created attachment 1603256 [details]
target libvirtd.log

The same question:
Migration fails with ceph storage domain which is mounted in local system

# mount -t ceph 10.73.224.204:6789:/  /mnt/cephfs/ -o name=admin,secret=AQCAgd5cFyMQLhAAHaz6w+WKy5LvmKjmRAViEg==

disk.xml:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/cephfs/qcow2.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>                                          
    </disk>

# virsh migrate demo qemu+ssh://10.16.56.165/system --live --verbose --p2p
setlocale: No such file or directory
error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /mnt/cephfs/qcow2.img which is already in use

Problem is that gluster (and I suspect ceph is the same) don't really support SELinux. They only emulate it locally. While they support XATTRs they don't really allow changing SELinux context. While it usually plays in our favor (e.g. migration is possible) with my recent work it plays against us. I mean, in case of dynamic SELinux labels (nearly every config I've seen, if not every single one), SELinux labels are generated at runtime by libvirt and they are random. So in general, the domain has different label on source than it has on the target. But since network filesystems don't really support SELinux (i.e. change in the label is not propagated to other hosts) - it is only locally emulated by kernel - migration works. The file shared on NFS/GlusterFS/Ceph/.. has two different labels (one from source POV the other from destination POV).
And here comes the problem with XATTRs - if a network FS supports XATTRs (GlusterFS and Ceph do) then XATTRs are set when domain is starting on the source and later, when destination tries to set its labels it finds mismatching record in XATTRs and migration is thus denied.
Looks like the best solution is to not store XATTRs if underlying FS does not support SELinux.

Patches posted upstream:

https://www.redhat.com/archives/libvir-list/2019-August/msg01041.html

Patches pushed upstream:

8fe953805a security_selinux: Play nicely with network FS that only emulates SELinux
eaa2a064fa security_selinux: Drop virSecuritySELinuxSetFileconHelper
b71d54f447 security_selinux: Drop @optional from _virSecuritySELinuxContextItem
079c1d6a29 security_selinux: Drop virSecuritySELinuxSetFileconOptional()
34712a5e3b virSecuritySELinuxSetFileconImpl: Drop @optional argument

v5.7.0-rc1-14-g8fe953805a

To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2019-September/msg00025.html

Verified on version:
libvirt-5.6.0-4.module+el8.1.0+4160+b50057dc.x86_64

Scenario 1:
Migrate with rbd backend:
1.
# mount -t ceph 10.73.224.204:6789:/  /mnt/cephfs/ -o name=admin,secret=AQCAgd5cFyMQLhAAHaz6w+WKy5LvmKjmRAViEg==  (both in source and target)
disk.xml:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/cephfs/qcow2.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>                                          
    </disk>

2.
# virsh migrate demo qemu+ssh://10.16.56.54/system --live --verbose --p2p
Migration: [100 %]
And then migrate back:
virsh migrate demo qemu+ssh://10.16.56.144/system --verbose --live --p2p
Migration: [100 %]

Scenario 2:
Migrate with gluster backend
1.Prepare an image in gluster storage and mount glusterfs on dst and src host
# mount -t glusterfs 10.66.4.119:/gvnew /mnt/gluster

2. Prepare a running VM whose image on the mount dir
# virsh domblklist demo
 Target   Source
--------------------------------------------------------
 hda      /mnt/gluster/test.qcow2

3.Start the domain and do migrate form 8.1 to 8.1
# virsh start demo                                                                                                                                                                             
Domain demo started 

# virsh migrate demo qemu+ssh://10.16.56.144/system --live --verbose --p2p --persistent 
Migration: [100 %]

And then migrate back:
virsh migrate demo --live qemu+ssh://10.16.56.54/system --verbose  --p2p --persistent 
Migration: [100 %]

Scenario 3:
1.Migrate with symlink dir
# mount -t glusterfs 10.66.4.119:/gvnew /mnt/gluster
# ln -s /mnt/gluster/ /var/lib/libvirt/images/glusterfs
# virsh domblklist demo
 Target   Source
-----------------------------------------------------
 vdb      /var/lib/libvirt/images/glusterfs/A.qcow2

2.
# virsh migrate demo --live qemu+ssh://10.16.56.54/system --verbose 
root.56.54's password: 
Migration: [100 %]


Work as expected

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723