Bug 1214846

Created attachment 1018065 [details]
Part of vdsm log with the error

Created attachment 1018066 [details]
sealert output

Created attachment 1018068 [details]
output of ps -efZ

To make sure this is selinux labeling issue, I did:

1. Set permissive mode:
   setenorce 0
2. Hotplug file-based disk
3. Use blockCopy to mirror the disk to block-based disk

The operation succeeds, and we can clearly see that the block device
is not using the correct selinux label:

# ls -lhZ /dev/dm-87
brw-rw----. vdsm qemu system_u:object_r:fixed_disk_device_t:s0 /dev/dm-87

And we can find new AVCs:

# ausearch -m avc

time->Thu Apr 23 12:38:38 2015
type=AVC msg=audit(1429807118.604:1136): avc:  denied  { read } for  pid=3963 comm="qemu-system-x86" name="dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 12:38:38 2015
type=AVC msg=audit(1429807118.604:1137): avc:  denied  { open } for  pid=3963 comm="qemu-system-x86" path="/dev/dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 12:38:38 2015
type=AVC msg=audit(1429807118.604:1138): avc:  denied  { ioctl } for  pid=3963 comm="qemu-system-x86" path="/dev/dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1

I tried the same using blockRebase:

    base = drive.diskReplicate["path"]
    flags = (libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY |
             libvirt.VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT |
             libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW)

    if drive.diskReplicate["diskType"] == DISK_TYPE.BLOCK:
        flags |= libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY_DEV

    self._dom.blockRebase(drive.name, base, flags=flags)

Thread-280::DEBUG::2015-04-23 12:58:08,559::vm::3604::vm.Vm::(_startDriveReplication) vmId=`a4397668-071a-4c15-836b-c49e04e1428d`::Replicating drive vdb to /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/13d6aaee-3668-4448-8580-e1d2f84f47d5/6cfd4d7d-f2ea-4fd0-9531-4c4239e983e1

It fails with the same error:

libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/13d6aaee-3668-4448-8580-e1d2f84f47d5/6cfd4d7d-f2ea-4fd0-9531-4c4239e983e1': Permission denied

And same AVC:

time->Thu Apr 23 12:58:08 2015
type=AVC msg=audit(1429808288.607:1671): avc:  denied  { read } for  pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
----
time->Thu Apr 23 12:58:08 2015
type=AVC msg=audit(1429808288.607:1672): avc:  denied  { read } for  pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
----
time->Thu Apr 23 12:58:08 2015
type=AVC msg=audit(1429808288.607:1673): avc:  denied  { read write } for  pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
----

And when using permissive mode, it works

Thread-882::DEBUG::2015-04-23 13:10:49,447::vm::3604::vm.Vm::(_startDriveReplication) vmId=`15189412-ee81-4a13-af63-60f6c013043f`::Replicating drive vda to /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/130e1480-ca62-413a-9f36-d32ae772b8ee/bd561729-60aa-483f-9575-c12e668a5e56

(this replication is successful)

The device is does not have selinux label:

# realpath /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/130e1480-ca62-413a-9f36-d32ae772b8ee/bd561729-60aa-483f-9575-c12e668a5e56
/dev/dm-12

# ls -Z /dev/dm-12
brw-rw----. vdsm qemu system_u:object_r:fixed_disk_device_t:s0 /dev/dm-12

And we see new AVCs:

time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2039): avc:  denied  { read } for  pid=21994 comm="qemu-system-x86" name="dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2040): avc:  denied  { open } for  pid=21994 comm="qemu-system-x86" path="/dev/dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2041): avc:  denied  { ioctl } for  pid=21994 comm="qemu-system-x86" path="/dev/dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2042): avc:  denied  { write } for  pid=21994 comm="qemu-system-x86" name="dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1

So this seems to be an issue with the underlying implementation that both
blockRebase and blockCopy are using.

It seems that 1.2.9.2 is missing this commit:

commit 7e130e8b3505ce0f821081dffde8c13a7ff921b3
Author: Peter Krempa <pkrempa>
Date:   Wed Nov 19 18:54:43 2014 +0100

    storage: qemu: Fix security labelling of new image chain elements
    
    When creating a disk image snapshot the libvirt code would blindly copy
    the parents label to the newly created image. This runs into problems
    when you start a VM from an image hosted on NFS (or other storage system
    that doesn't support selinux labels) and the snapshot destination is on
    a storage system that does support selinux labels. Libvirt's code in
    that case generates a different security label for the image hosted on
    NFS. This label is valid only for NFS images and doesn't allow access in
    case of a locally stored image.
    
    To fix this issue libvirt needs to refrain from copying security
    information in cases where the default domain seclabel is a better
    choice.
    
    This patch repurposes the now unused @force argument of
    virStorageSourceInitChainElement to denote whether a copy of the
    security labelling stuff should be attempted or not. This allows to
    fine-control the copy operation for cases where we need to keep the
    label of the old disk vs. the cases where we need to keep the label
    unset to use the default domain imagelabel.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1151718

This commit was backported to rhel in:

commit d4b88a0c0def14366ef18bd6ab18689f0bbd18f7
Author: Peter Krempa <pkrempa>
Date:   Fri Nov 21 09:53:04 2014 +0100

And on rhel (libvirt-daemon-1.2.8-16.el7_1.3), blockRebae does label
the block device.

I backported 7e130e8b3505ce0f821081dffde8c13a7ff921b3 on top of v1.2.9-maint
a run a quick test, and the issue seems to be resolved.

Tested flows
- mirror from file to block
- mirror from block to file
- mirror from file to file
- mirror from block to block

All tests done with vm without a guest os, copying empty disks around.

libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21

Package libvirt-1.2.9.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21
then log in and leave karma (feedback).

Is it possible to get this update for fedora 20 virt-preview repo?

(In reply to Nir Soffer from comment #10)
> Is it possible to get this update for fedora 20 virt-preview repo?

No, I'm sorry. fedora20 virt-preview hasn't been maintained for months, and f20 is EOL in two months anyways.