Bug 1214846 - virDomainBlockCopy or virDomainBlockRebase fails with Permission denied when copying from file to block device
Summary: virDomainBlockCopy or virDomainBlockRebase fails with Permission denied when ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-04-23 16:17 UTC by Nir Soffer
Modified: 2015-06-05 23:02 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-05 23:02:29 UTC
Type: Bug


Attachments (Terms of Use)
Part of vdsm log with the error (4.40 KB, text/plain)
2015-04-23 16:21 UTC, Nir Soffer
no flags Details
sealert output (3.99 KB, text/plain)
2015-04-23 16:24 UTC, Nir Soffer
no flags Details
output of ps -efZ (3.66 KB, text/plain)
2015-04-23 16:28 UTC, Nir Soffer
no flags Details

Description Nir Soffer 2015-04-23 16:17:31 UTC
Description of problem:

When using virDomainBlockCopy to mirror file disk to block device, blockCopy
fails with:

libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/f97b24b4-c52c-4691-94d0-54118970b050/67bc56a1-c873-4138-b047-1e1b6873d1e2': Permission denied

This is the destxml argument to blockCopy:

<disk device="disk" snapshot="no" type="block">
        <source dev="/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/f97b24b4-c52c-4691-94d0-54118970b050/67bc56a1-c873-4138-b047-1e1b6873d1e2"/>
        <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/>
</disk>

Vdsm creates this block device (lv), and create a symlink (/rhev/data-center...) pointing to it.

And the flags are:

flags = (libvirt.VIR_DOMAIN_BLOCK_COPY_SHALLOW |
         libvirt.VIR_DOMAIN_BLOCK_COPY_REUSE_EXT)

Checking audit.log, we see that selinux is preventing access to the
block device:

--------------------------------------------------------------------------------

SELinux is preventing qemu-system-x86 from 'read, write' accesses on the blk_file dm-87.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that qemu-system-x86 should be allowed read write access on the dm-87 blk_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep qemu-system-x86 /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp


Additional Information:
Source Context                system_u:system_r:svirt_t:s0:c418,c961
Target Context                system_u:object_r:fixed_disk_device_t:s0
Target Objects                dm-87 [ blk_file ]
Source                        qemu-system-x86
Source Path                   qemu-system-x86
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-105.13.fc21.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     voodoo7.tlv.redhat.com
Platform                      Linux voodoo7.tlv.redhat.com
                              3.19.4-200.fc21.x86_64 #1 SMP Mon Apr 13 21:43:26
                              UTC 2015 x86_64 x86_64
Alert Count                   1
First Seen                    2015-04-23 11:48:04 EDT
Last Seen                     2015-04-23 11:48:04 EDT
Local ID                      2df74b48-391f-4aa3-8d7d-6173c2ce73d6

Raw Audit Messages
type=AVC msg=audit(1429804084.669:562): avc:  denied  { read write } for  pid=3963 comm="qemu-system-x86" name="dm-87" dev="devtmpfs" ino=110225 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0

Hash: qemu-system-x86,svirt_t,fixed_disk_device_t,blk_file,read,write


Version-Release number of selected component (if applicable):

# rpm -qa | grep libvirt
libvirt-daemon-driver-qemu-1.2.9.2-1.fc21.x86_64
libvirt-daemon-driver-storage-1.2.9.2-1.fc21.x86_64
libvirt-client-1.2.9.2-1.fc21.x86_64
libvirt-daemon-kvm-1.2.9.2-1.fc21.x86_64
libvirt-daemon-1.2.9.2-1.fc21.x86_64
libvirt-python-1.2.9-2.fc21.x86_64
libvirt-daemon-driver-nodedev-1.2.9.2-1.fc21.x86_64
libvirt-daemon-driver-network-1.2.9.2-1.fc21.x86_64
libvirt-daemon-config-nwfilter-1.2.9.2-1.fc21.x86_64
libvirt-daemon-driver-secret-1.2.9.2-1.fc21.x86_64
libvirt-daemon-driver-interface-1.2.9.2-1.fc21.x86_64
libvirt-daemon-driver-nwfilter-1.2.9.2-1.fc21.x86_64
libvirt-lock-sanlock-1.2.9.2-1.fc21.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Start vm with file based disk
2. Try to use blockCopy to mirror drive to block device

Workaround:
Use permissive mode.

Additional info:

This looks like a labeling issue in libvirt. When hot-plugging a block
based disk to this vm, libvirt lebal the disk as svirt_image_t:s0:c418,c961,
allowing the underlying qemu process to access the device:

# ls -Z /dev/dm-86
brw-rw----. vdsm qemu system_u:object_r:svirt_image_t:s0:c418,c961 /dev/dm-86

But when qemu try to access the block device, it fails because it is using the
default selinux label: system_u:object_r:fixed_disk_device_t:s0

In the past we had a bug were udev was removing libvirt selinux labels after
refreshing a lv, but we believe this issue is fixed.

Comment 1 Nir Soffer 2015-04-23 16:21:23 UTC
Created attachment 1018065 [details]
Part of vdsm log with the error

Comment 2 Nir Soffer 2015-04-23 16:24:00 UTC
Created attachment 1018066 [details]
sealert output

Comment 3 Nir Soffer 2015-04-23 16:28:24 UTC
Created attachment 1018068 [details]
output of ps -efZ

Comment 4 Nir Soffer 2015-04-23 16:49:37 UTC
To make sure this is selinux labeling issue, I did:

1. Set permissive mode:
   setenorce 0
2. Hotplug file-based disk
3. Use blockCopy to mirror the disk to block-based disk

The operation succeeds, and we can clearly see that the block device
is not using the correct selinux label:

# ls -lhZ /dev/dm-87
brw-rw----. vdsm qemu system_u:object_r:fixed_disk_device_t:s0 /dev/dm-87

And we can find new AVCs:

# ausearch -m avc

time->Thu Apr 23 12:38:38 2015
type=AVC msg=audit(1429807118.604:1136): avc:  denied  { read } for  pid=3963 comm="qemu-system-x86" name="dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 12:38:38 2015
type=AVC msg=audit(1429807118.604:1137): avc:  denied  { open } for  pid=3963 comm="qemu-system-x86" path="/dev/dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 12:38:38 2015
type=AVC msg=audit(1429807118.604:1138): avc:  denied  { ioctl } for  pid=3963 comm="qemu-system-x86" path="/dev/dm-86" dev="devtmpfs" ino=372160 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1

Comment 5 Nir Soffer 2015-04-23 17:24:34 UTC
I tried the same using blockRebase:

    base = drive.diskReplicate["path"]
    flags = (libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY |
             libvirt.VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT |
             libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW)

    if drive.diskReplicate["diskType"] == DISK_TYPE.BLOCK:
        flags |= libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY_DEV

    self._dom.blockRebase(drive.name, base, flags=flags)

Thread-280::DEBUG::2015-04-23 12:58:08,559::vm::3604::vm.Vm::(_startDriveReplication) vmId=`a4397668-071a-4c15-836b-c49e04e1428d`::Replicating drive vdb to /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/13d6aaee-3668-4448-8580-e1d2f84f47d5/6cfd4d7d-f2ea-4fd0-9531-4c4239e983e1

It fails with the same error:

libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/13d6aaee-3668-4448-8580-e1d2f84f47d5/6cfd4d7d-f2ea-4fd0-9531-4c4239e983e1': Permission denied

And same AVC:

time->Thu Apr 23 12:58:08 2015
type=AVC msg=audit(1429808288.607:1671): avc:  denied  { read } for  pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
----
time->Thu Apr 23 12:58:08 2015
type=AVC msg=audit(1429808288.607:1672): avc:  denied  { read } for  pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
----
time->Thu Apr 23 12:58:08 2015
type=AVC msg=audit(1429808288.607:1673): avc:  denied  { read write } for  pid=3963 comm="qemu-system-x86" name="dm-11" dev="devtmpfs" ino=489112 scontext=system_u:system_r:svirt_t:s0:c418,c961 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
----

And when using permissive mode, it works

Thread-882::DEBUG::2015-04-23 13:10:49,447::vm::3604::vm.Vm::(_startDriveReplication) vmId=`15189412-ee81-4a13-af63-60f6c013043f`::Replicating drive vda to /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/130e1480-ca62-413a-9f36-d32ae772b8ee/bd561729-60aa-483f-9575-c12e668a5e56

(this replication is successful)

The device is does not have selinux label:

# realpath /rhev/data-center/cca1d436-8c16-4887-8c07-2a9ecc1c0830/661f1926-04cc-405a-85eb-802563b48ed3/images/130e1480-ca62-413a-9f36-d32ae772b8ee/bd561729-60aa-483f-9575-c12e668a5e56
/dev/dm-12

# ls -Z /dev/dm-12
brw-rw----. vdsm qemu system_u:object_r:fixed_disk_device_t:s0 /dev/dm-12

And we see new AVCs:

time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2039): avc:  denied  { read } for  pid=21994 comm="qemu-system-x86" name="dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2040): avc:  denied  { open } for  pid=21994 comm="qemu-system-x86" path="/dev/dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2041): avc:  denied  { ioctl } for  pid=21994 comm="qemu-system-x86" path="/dev/dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1
----
time->Thu Apr 23 13:10:49 2015
type=AVC msg=audit(1429809049.454:2042): avc:  denied  { write } for  pid=21994 comm="qemu-system-x86" name="dm-12" dev="devtmpfs" ino=571988 scontext=system_u:system_r:svirt_t:s0:c425,c690 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=1

So this seems to be an issue with the underlying implementation that both
blockRebase and blockCopy are using.

Comment 6 Nir Soffer 2015-04-23 20:01:12 UTC
It seems that 1.2.9.2 is missing this commit:

commit 7e130e8b3505ce0f821081dffde8c13a7ff921b3
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Nov 19 18:54:43 2014 +0100

    storage: qemu: Fix security labelling of new image chain elements
    
    When creating a disk image snapshot the libvirt code would blindly copy
    the parents label to the newly created image. This runs into problems
    when you start a VM from an image hosted on NFS (or other storage system
    that doesn't support selinux labels) and the snapshot destination is on
    a storage system that does support selinux labels. Libvirt's code in
    that case generates a different security label for the image hosted on
    NFS. This label is valid only for NFS images and doesn't allow access in
    case of a locally stored image.
    
    To fix this issue libvirt needs to refrain from copying security
    information in cases where the default domain seclabel is a better
    choice.
    
    This patch repurposes the now unused @force argument of
    virStorageSourceInitChainElement to denote whether a copy of the
    security labelling stuff should be attempted or not. This allows to
    fine-control the copy operation for cases where we need to keep the
    label of the old disk vs. the cases where we need to keep the label
    unset to use the default domain imagelabel.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1151718

This commit was backported to rhel in:

commit d4b88a0c0def14366ef18bd6ab18689f0bbd18f7
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Fri Nov 21 09:53:04 2014 +0100

And on rhel (libvirt-daemon-1.2.8-16.el7_1.3), blockRebae does label
the block device.

Comment 7 Nir Soffer 2015-04-23 22:08:49 UTC
I backported 7e130e8b3505ce0f821081dffde8c13a7ff921b3 on top of v1.2.9-maint
a run a quick test, and the issue seems to be resolved.

Tested flows
- mirror from file to block
- mirror from block to file
- mirror from file to file
- mirror from block to block

All tests done with vm without a guest os, copying empty disks around.

Comment 8 Fedora Update System 2015-04-28 16:55:01 UTC
libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21

Comment 9 Fedora Update System 2015-04-29 13:06:10 UTC
Package libvirt-1.2.9.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21
then log in and leave karma (feedback).

Comment 10 Nir Soffer 2015-04-30 21:58:25 UTC
Is it possible to get this update for fedora 20 virt-preview repo?

Comment 11 Cole Robinson 2015-04-30 23:10:18 UTC
(In reply to Nir Soffer from comment #10)
> Is it possible to get this update for fedora 20 virt-preview repo?

No, I'm sorry. fedora20 virt-preview hasn't been maintained for months, and f20 is EOL in two months anyways.


Note You need to log in before you can comment on or make changes to this bug.