Description of problem: When using virDomainBlockCopy to mirror file disk to block device, blockCopy fails with: libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/f97b24b4-c52c-4691-94d0-54118970b050/67bc56a1-c873-4138-b047-1e1b6873d1e2': Permission denied This is the destxml argument to blockCopy: <disk device="disk" snapshot="no" type="block"> <source dev="/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/f97b24b4-c52c-4691-94d0-54118970b050/67bc56a1-c873-4138-b047-1e1b6873d1e2"/> <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/> </disk> Vdsm creates this block device (lv), and create a symlink (/rhev/data-center...) pointing to it. And the flags are: flags = (libvirt.VIR_DOMAIN_BLOCK_COPY_SHALLOW | libvirt.VIR_DOMAIN_BLOCK_COPY_REUSE_EXT) Checking audit.log, we see that selinux is preventing access to the block device: -------------------------------------------------------------------------------- SELinux is preventing qemu-system-x86 from 'read, write' accesses on the blk_file dm-87. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that qemu-system-x86 should be allowed read write access on the dm-87 blk_file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep qemu-system-x86 /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:svirt_t:s0:c418,c961 Target Context system_u:object_r:fixed_disk_device_t:s0 Target Objects dm-87 [ blk_file ] Source qemu-system-x86 Source Path qemu-system-x86 Port <Unknown> Host <Unknown> Source RPM Packages Target RPM Packages Policy RPM selinux-policy-3.13.1-105.13.fc21.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name voodoo7.tlv.redhat.com Platform Linux voodoo7.tlv.redhat.com 3.19.4-200.fc21.x86_64 #1 SMP Mon Apr 13 21:43:26 UTC 2015 x86_64 x86_64 Alert Count 1 First Seen 2015-04-23 11:48:04 EDT Last Seen 2015-04-23 11:48:04 EDT Local ID 2df74b48-391f-4aa3-8d7d-6173c2ce73d6 Raw Audit Messages type=AVC msg=audit(1429804084.669:562): avc: denied { read write } for pid=3963 comm="qemu-system-x86" name="dm-87" dev="devtmpfs" ino=110225 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0 Hash: qemu-system-x86,svirt_t,fixed_disk_device_t,blk_file,read,write Version-Release number of selected component (if applicable): # rpm -qa | grep libvirt libvirt-daemon-driver-qemu-1.2.9.2-1.fc21.x86_64 libvirt-daemon-driver-storage-1.2.9.2-1.fc21.x86_64 libvirt-client-1.2.9.2-1.fc21.x86_64 libvirt-daemon-kvm-1.2.9.2-1.fc21.x86_64 libvirt-daemon-1.2.9.2-1.fc21.x86_64 libvirt-python-1.2.9-2.fc21.x86_64 libvirt-daemon-driver-nodedev-1.2.9.2-1.fc21.x86_64 libvirt-daemon-driver-network-1.2.9.2-1.fc21.x86_64 libvirt-daemon-config-nwfilter-1.2.9.2-1.fc21.x86_64 libvirt-daemon-driver-secret-1.2.9.2-1.fc21.x86_64 libvirt-daemon-driver-interface-1.2.9.2-1.fc21.x86_64 libvirt-daemon-driver-nwfilter-1.2.9.2-1.fc21.x86_64 libvirt-lock-sanlock-1.2.9.2-1.fc21.x86_64 How reproducible: Always Steps to Reproduce: 1. Start vm with file based disk 2. Try to use blockCopy to mirror drive to block device Workaround: Use permissive mode. Additional info: This looks like a labeling issue in libvirt. When hot-plugging a block based disk to this vm, libvirt lebal the disk as svirt_image_t:s0:c418,c961, allowing the underlying qemu process to access the device: # ls -Z /dev/dm-86 brw-rw----. vdsm qemu system_u:object_r:svirt_image_t:s0:c418,c961 /dev/dm-86 But when qemu try to access the block device, it fails because it is using the default selinux label: system_u:object_r:fixed_disk_device_t:s0 In the past we had a bug were udev was removing libvirt selinux labels after refreshing a lv, but we believe this issue is fixed.
Created attachment 1018065 [details] Part of vdsm log with the error
Created attachment 1018066 [details] sealert output
Created attachment 1018068 [details] output of ps -efZ
To make sure this is selinux labeling issue, I did: 1. Set permissive mode: setenorce 0 2. Hotplug file-based disk 3. Use blockCopy to mirror the disk to block-based disk The operation succeeds, and we can clearly see that the block device is not using the correct selinux label: # ls -lhZ /dev/dm-87 brw-rw----. vdsm qemu system_u:object_r:fixed_disk_device_t:s0 /dev/dm-87 And we can find new AVCs: # ausearch -m avc time->Thu Apr 23 12:38:38 2015 type=AVC msg=audit(1429807118.604:1136): avc: denied { read } for pid=3963 comm="qemu-system-x86" name="dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1 ---- time->Thu Apr 23 12:38:38 2015 type=AVC msg=audit(1429807118.604:1137): avc: denied { open } for pid=3963 comm="qemu-system-x86" path="/dev/dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1 ---- time->Thu Apr 23 12:38:38 2015 type=AVC msg=audit(1429807118.604:1138): avc: denied { ioctl } for pid=3963 comm="qemu-system-x86" path="/dev/dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
I tried the same using blockRebase: base = drive.diskReplicate["path"] flags = (libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY | libvirt.VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT | libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW) if drive.diskReplicate["diskType"] == DISK_TYPE.BLOCK: flags |= libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY_DEV self._dom.blockRebase(drive.name, base, flags=flags) Thread-280::DEBUG::2015-04-23 12:58:08,559::vm::3604::vm.Vm::(_startDriveReplication) vmId=`a4397668-071a-4c15-836b-c49e04e1428d`::Replicating drive vdb to /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/13d6aaee-3668-4448-8580-e1d2f84f47d5/6cfd4d7d-f2ea-4fd0-9531-4c4239e983e1 It fails with the same error: libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/13d6aaee-3668-4448-8580-e1d2f84f47d5/6cfd4d7d-f2ea-4fd0-9531-4c4239e983e1': Permission denied And same AVC: time->Thu Apr 23 12:58:08 2015 type=AVC msg=audit(1429808288.607:1671): avc: denied { read } for pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0 ---- time->Thu Apr 23 12:58:08 2015 type=AVC msg=audit(1429808288.607:1672): avc: denied { read } for pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0 ---- time->Thu Apr 23 12:58:08 2015 type=AVC msg=audit(1429808288.607:1673): avc: denied { read write } for pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0 ---- And when using permissive mode, it works Thread-882::DEBUG::2015-04-23 13:10:49,447::vm::3604::vm.Vm::(_startDriveReplication) vmId=`15189412-ee81-4a13-af63-60f6c013043f`::Replicating drive vda to /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/130e1480-ca62-413a-9f36-d32ae772b8ee/bd561729-60aa-483f-9575-c12e668a5e56 (this replication is successful) The device is does not have selinux label: # realpath /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/130e1480-ca62-413a-9f36-d32ae772b8ee/bd561729-60aa-483f-9575-c12e668a5e56 /dev/dm-12 # ls -Z /dev/dm-12 brw-rw----. vdsm qemu system_u:object_r:fixed_disk_device_t:s0 /dev/dm-12 And we see new AVCs: time->Thu Apr 23 13:10:49 2015 type=AVC msg=audit(1429809049.454:2039): avc: denied { read } for pid=21994 comm="qemu-system-x86" name="dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1 ---- time->Thu Apr 23 13:10:49 2015 type=AVC msg=audit(1429809049.454:2040): avc: denied { open } for pid=21994 comm="qemu-system-x86" path="/dev/dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1 ---- time->Thu Apr 23 13:10:49 2015 type=AVC msg=audit(1429809049.454:2041): avc: denied { ioctl } for pid=21994 comm="qemu-system-x86" path="/dev/dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1 ---- time->Thu Apr 23 13:10:49 2015 type=AVC msg=audit(1429809049.454:2042): avc: denied { write } for pid=21994 comm="qemu-system-x86" name="dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1 So this seems to be an issue with the underlying implementation that both blockRebase and blockCopy are using.
It seems that 1.2.9.2 is missing this commit: commit 7e130e8b3505ce0f821081dffde8c13a7ff921b3 Author: Peter Krempa <pkrempa> Date: Wed Nov 19 18:54:43 2014 +0100 storage: qemu: Fix security labelling of new image chain elements When creating a disk image snapshot the libvirt code would blindly copy the parents label to the newly created image. This runs into problems when you start a VM from an image hosted on NFS (or other storage system that doesn't support selinux labels) and the snapshot destination is on a storage system that does support selinux labels. Libvirt's code in that case generates a different security label for the image hosted on NFS. This label is valid only for NFS images and doesn't allow access in case of a locally stored image. To fix this issue libvirt needs to refrain from copying security information in cases where the default domain seclabel is a better choice. This patch repurposes the now unused @force argument of virStorageSourceInitChainElement to denote whether a copy of the security labelling stuff should be attempted or not. This allows to fine-control the copy operation for cases where we need to keep the label of the old disk vs. the cases where we need to keep the label unset to use the default domain imagelabel. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1151718 This commit was backported to rhel in: commit d4b88a0c0def14366ef18bd6ab18689f0bbd18f7 Author: Peter Krempa <pkrempa> Date: Fri Nov 21 09:53:04 2014 +0100 And on rhel (libvirt-daemon-1.2.8-16.el7_1.3), blockRebae does label the block device.
I backported 7e130e8b3505ce0f821081dffde8c13a7ff921b3 on top of v1.2.9-maint a run a quick test, and the issue seems to be resolved. Tested flows - mirror from file to block - mirror from block to file - mirror from file to file - mirror from block to block All tests done with vm without a guest os, copying empty disks around.
libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21
Package libvirt-1.2.9.3-1.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21 then log in and leave karma (feedback).
Is it possible to get this update for fedora 20 virt-preview repo?
(In reply to Nir Soffer from comment #10) > Is it possible to get this update for fedora 20 virt-preview repo? No, I'm sorry. fedora20 virt-preview hasn't been maintained for months, and f20 is EOL in two months anyways.