RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1151718 - Permission denied when create external snapshot for guest whose source file based on nfs
Summary: Permission denied when create external snapshot for guest whose source file b...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-11 07:30 UTC by Shanzhi Yu
Modified: 2015-03-05 07:46 UTC (History)
6 users (show)

Fixed In Version: libvirt-1.2.8-8.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-05 07:46:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
libvirtd log (202.81 KB, text/plain)
2014-10-13 08:29 UTC, Shanzhi Yu 		no flags Details
logoflibvirtd (843.26 KB, text/plain)
2014-11-28 08:13 UTC, Shanzhi Yu 		no flags Details
logofaudit (1.36 MB, text/plain)
2014-11-28 08:14 UTC, Shanzhi Yu 		no flags Details
Log of Libvirtd (205.45 KB, text/plain)
2014-12-02 03:12 UTC, Shanzhi Yu 		no flags Details
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:0323 0 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2015-03-05 12:10:54 UTC

Description Shanzhi Yu 2014-10-11 07:30:17 UTC 
Description of problem:

Permission denied when create external snapshot for guest whose source file based on nfs 

Version-Release number of selected component (if applicable):

libvirt-1.2.8-5.el7.x86_64
qemu-kvm-rhev-1.5.3-60.el7_0.10.x86_64
selinux-policy-3.13.1-4.el7.noarch

How reproducible:

100%

Steps to Reproduce:

1. Prepare a guest with source file based on NFS server

# getenforce && getsebool virt_use_nfs

Enforcing
virt_use_nfs --> on

#mount -o soft 10.66.x.xxx:/mnt/nfs /mnt/

# mount |grep nfs 
10.66.x.xxx:/mnt/nfs on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.66.5.226,local_lock=none,addr=10.66.x.xxx)

# virsh domstate rhel6;virsh domblklist rhel6 
running

Target     Source
------------------------------------------------
vda        /mnt/kvm-rhel6.5-x86_64-qcow2.img


2. create external snapshot with snapshot file base on local
#virsh snapshot-create-as rhel6 s1 --diskspce vda,file=/var/lib/libvirt/images/rhel6.s1 --disk-only 
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied



Actual results:


Expected results:


Additional info:

disable selinux will succeed in step 2

# egrep "error|execute" /tmp/libvirtd.log  

2014-09-09 05:21:55.906+0000: 6328: debug : virJSONValueToString:1303 : result={"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-virtio-disk0","snapshot-file":"/var/lib/libvirt/images/rhel6.s1","format":"qcow2"}}]},"id":"libvirt-13"}
2014-09-09 05:21:55.906+0000: 6328: debug : qemuMonitorJSONCommandWithFd:286 : Send command '{"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-virtio-disk0","snapshot-file":"/var/lib/libvirt/images/rhel6.s1","format":"qcow2"}}]},"id":"libvirt-13"}' for write with FD -1
2014-09-09 05:21:55.906+0000: 6328: debug : qemuMonitorSend:969 : QEMU_MONITOR_SEND_MSG: mon=0x7f04080817d0 msg={"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-virtio-disk0","snapshot-file":"/var/lib/libvirt/images/rhel6.s1","format":"qcow2"}}]},"id":"libvirt-13"}
2014-09-09 05:21:55.907+0000: 6325: debug : qemuMonitorIOWrite:507 : QEMU_MONITOR_IO_WRITE: mon=0x7f04080817d0 buf={"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-virtio-disk0","snapshot-file":"/var/lib/libvirt/images/rhel6.s1","format":"qcow2"}}]},"id":"libvirt-13"}
2014-09-09 05:21:55.917+0000: 6325: debug : qemuMonitorIOProcess:399 : QEMU_MONITOR_IO_PROCESS: mon=0x7f04080817d0 buf={"id": "libvirt-13", "error": {"class": "GenericError", "desc": "Could not create file: Permission denied"}}
2014-09-09 05:21:55.917+0000: 6325: debug : qemuMonitorJSONIOProcessLine:179 : Line [{"id": "libvirt-13", "error": {"class": "GenericError", "desc": "Could not create file: Permission denied"}}]
2014-09-09 05:21:55.917+0000: 6325: debug : virJSONValueFromString:1136 : string={"id": "libvirt-13", "error": {"class": "GenericError", "desc": "Could not create file: Permission denied"}}
2014-09-09 05:21:55.917+0000: 6325: debug : qemuMonitorJSONIOProcessLine:199 : QEMU_MONITOR_RECV_REPLY: mon=0x7f04080817d0 reply={"id": "libvirt-13", "error": {"class": "GenericError", "desc": "Could not create file: Permission denied"}}
2014-09-09 05:21:55.918+0000: 6328: debug : virJSONValueToString:1303 : result={"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-virtio-disk0","snapshot-file":"/var/lib/libvirt/images/rhel6.s1","format":"qcow2"}}]},"id":"libvirt-13"}
2014-09-09 05:21:55.918+0000: 6328: debug : virJSONValueToString:1303 : result={"id":"libvirt-13","error":{"class":"GenericError","desc":"Could not create file: Permission denied"}}
2014-09-09 05:21:55.918+0000: 6328: debug : qemuMonitorJSONCheckError:366 : unable to execute QEMU command {"execute":"transaction","arguments":{"actions":[{"type":"blockdev-snapshot-sync","data":{"device":"drive-virtio-disk0","snapshot-file":"/var/lib/libvirt/images/rhel6.s1","format":"qcow2"}}]},"id":"libvirt-13"}: {"id":"libvirt-13","error":{"class":"GenericError","desc":"Could not create file: Permission denied"}}
2014-09-09 05:21:55.918+0000: 6328: error : qemuMonitorJSONCheckError:377 : internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied
Comment 1 Peter Krempa 2014-10-12 08:07:51 UTC 
Could you please attach the full debug log.
Comment 2 Shanzhi Yu 2014-10-13 08:29:36 UTC 
Created attachment 946274 [details]
libvirtd log
Comment 3 Peter Krempa 2014-10-29 10:24:54 UTC 
I'm suspecting that the denial may be caused by selinux. Could you please look into the audit log for a possible clue?.
Comment 4 Shanzhi Yu 2014-11-11 04:53:49 UTC 
Yeah, it should be selinux's problem


# virsh snapshot-create-as rhel6 s1 --diskspec vda,file=/var/lib/libvirt/images/rhel6.s1 --disk-only 
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

# ausearch -m avc -ts recent
----
time->Tue Nov 11 12:50:21 2014
type=SYSCALL msg=audit(1415681421.162:7324): arch=c000003e syscall=2 success=no exit=-13 a0=7f8fffd360b0 a1=80800 a2=0 a3=7f8ff6d10020 items=0 ppid=1 pid=14759 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c25,c329 key=(null)
type=AVC msg=audit(1415681421.162:7324): avc:  denied  { read } for  pid=14759 comm="qemu-kvm" name="rhel6.s1" dev="sda1" ino=1707175 scontext=system_u:system_r:svirt_t:s0:c25,c329 tcontext=system_u:object_r:virt_image_t:s0 tclass=file
----
time->Tue Nov 11 12:50:21 2014
type=SYSCALL msg=audit(1415681421.162:7325): arch=c000003e syscall=2 success=no exit=-13 a0=7f8fffd360b0 a1=80800 a2=0 a3=7f8ff6d10020 items=0 ppid=1 pid=14759 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c25,c329 key=(null)
type=AVC msg=audit(1415681421.162:7325): avc:  denied  { read } for  pid=14759 comm="qemu-kvm" name="rhel6.s1" dev="sda1" ino=1707175 scontext=system_u:system_r:svirt_t:s0:c25,c329 tcontext=system_u:object_r:virt_image_t:s0 tclass=file
----
time->Tue Nov 11 12:50:21 2014
type=SYSCALL msg=audit(1415681421.168:7326): arch=c000003e syscall=2 success=no exit=-13 a0=7f8fffdba4d0 a1=80800 a2=0 a3=7f8ff6d10de0 items=0 ppid=1 pid=14759 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c25,c329 key=(null)
type=AVC msg=audit(1415681421.168:7326): avc:  denied  { read } for  pid=14759 comm="qemu-kvm" name="rhel6.s1" dev="sda1" ino=1707175 scontext=system_u:system_r:svirt_t:s0:c25,c329 tcontext=system_u:object_r:virt_image_t:s0 tclass=file
----
time->Tue Nov 11 12:50:21 2014
type=SYSCALL msg=audit(1415681421.168:7327): arch=c000003e syscall=2 success=no exit=-13 a0=7f8fffdba4d0 a1=80800 a2=0 a3=7f8ff6d10de0 items=0 ppid=1 pid=14759 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c25,c329 key=(null)
type=AVC msg=audit(1415681421.168:7327): avc:  denied  { read } for  pid=14759 comm="qemu-kvm" name="rhel6.s1" dev="sda1" ino=1707175 scontext=system_u:system_r:svirt_t:s0:c25,c329 tcontext=system_u:object_r:virt_image_t:s0 tclass=file
----
time->Tue Nov 11 12:50:21 2014
type=SYSCALL msg=audit(1415681421.168:7328): arch=c000003e syscall=2 success=no exit=-13 a0=7f8fffdba460 a1=80241 a2=1a4 a3=7f8ff6d107a0 items=0 ppid=1 pid=14759 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c25,c329 key=(null)
type=AVC msg=audit(1415681421.168:7328): avc:  denied  { write } for  pid=14759 comm="qemu-kvm" name="rhel6.s1" dev="sda1" ino=1707175 scontext=system_u:system_r:svirt_t:s0:c25,c329 tcontext=system_u:object_r:virt_image_t:s0 tclass=file
Comment 5 Peter Krempa 2014-11-21 08:35:12 UTC 
Fixed upstream:

commit 7e130e8b3505ce0f821081dffde8c13a7ff921b3
Author: Peter Krempa <pkrempa>
Date:   Wed Nov 19 18:54:43 2014 +0100

    storage: qemu: Fix security labelling of new image chain elements
    
    When creating a disk image snapshot the libvirt code would blindly copy
    the parents label to the newly created image. This runs into problems
    when you start a VM from an image hosted on NFS (or other storage system
    that doesn't support selinux labels) and the snapshot destination is on
    a storage system that does support selinux labels. Libvirt's code in
    that case generates a different security label for the image hosted on
    NFS. This label is valid only for NFS images and doesn't allow access in
    case of a locally stored image.
    
    To fix this issue libvirt needs to refrain from copying security
    information in cases where the default domain seclabel is a better
    choice.
    
    This patch repurposes the now unused @force argument of
    virStorageSourceInitChainElement to denote whether a copy of the
    security labelling stuff should be attempted or not. This allows to
    fine-control the copy operation for cases where we need to keep the
    label of the old disk vs. the cases where we need to keep the label
    unset to use the default domain imagelabel.
Comment 8 Shanzhi Yu 2014-11-24 07:41:06 UTC 
I can't verify this bug, since there is other failed case.

If guest source file is based on NFS, then create external snapshot with new snapshot file also be based on NFS, will met permission error.

1.

# getenforce && getsebool virt_use_nfs
Enforcing
virt_use_nfs --> on


2. mount nfs to local

# mount -o soft 10.66.x.xxx:/mnt/nfs /mnt/

3. start guest with source file based on nfs

# virsh domblklist r7 ;virsh start r7 
Target     Source
------------------------------------------------
vda        /mnt/rhel6.img

Domain r7 started

4. create external snapshot with new file also based on nfs

# virsh snapshot-create-as r7 ss --disk-only 
error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

5. create external snapshot with new file based on local

# virsh snapshot-create-as r7 ss --disk-only --diskspec vda,file=/var/lib/libvirt/images/r7.ss 
Domain snapshot ss created

# virsh dumpxml r7|grep disk -A 8
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/r7.ss'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/mnt/rhel6.img'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
Comment 9 Peter Krempa 2014-11-25 13:41:06 UTC 
(In reply to Shanzhi Yu from comment #8)
> I can't verify this bug, since there is other failed case.
> 
> If guest source file is based on NFS, then create external snapshot with new
> snapshot file also be based on NFS, will met permission error.
> 
> 1.
> 
> # getenforce && getsebool virt_use_nfs
> Enforcing
> virt_use_nfs --> on
> 
> 
> 2. mount nfs to local
> 
> # mount -o soft 10.66.x.xxx:/mnt/nfs /mnt/
> 
> 3. start guest with source file based on nfs
> 
> # virsh domblklist r7 ;virsh start r7 
> Target     Source
> ------------------------------------------------
> vda        /mnt/rhel6.img
> 
> Domain r7 started
> 
> 4. create external snapshot with new file also based on nfs
> 
> # virsh snapshot-create-as r7 ss --disk-only 
> error: internal error: unable to execute QEMU command 'transaction': Could
> not create file: Permission denied
> 

I can't reproduce that by the steps you've described here. Could you please attach the following info:
1) libvirtd debug log
2) selinux audit log
3) mount options of the mount point (export)
4) ls -lia of the mount point
Comment 10 Shanzhi Yu 2014-11-28 07:57:25 UTC 
1. libvirtd log is attached

2. selinux audit log

# ausearch -m avc -ts recent
<no matches>

audit.log also be attached.

3. mount options
3.1 on nfs server
# cat /etc/exports
#/mnt/nfs *(async,rw,all_squash,anonuid=107,anonuid=107)
# *(rw,no_root_squash)
/mnt/nfs *(rw)

3.2 on client

# mount|grep /mnt
10.66.6.111:/mnt/nfs on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.66.4.110,local_lock=none,addr=10.66.6.111)

4. ls mount
# ls -lia /mnt/
total 5042244
2237486 drwxrwxrwx.  2 root      root            4096 Nov 28 15:41 .
      2 dr-xr-xr-x. 20 root      root            4096 Nov 27 18:03 ..
2237487 -rwxrwxrwx.  1 qemu      qemu      3482714112 Nov 28 15:44 rhel6.img
2259106 -rw-------.  1 qemu      nfsnobody  393216000 Oct 15 14:53 rhel6.t2
Comment 11 Shanzhi Yu 2014-11-28 08:13:48 UTC 
Created attachment 962397 [details]
logoflibvirtd
Comment 12 Shanzhi Yu 2014-11-28 08:14:25 UTC 
Created attachment 962398 [details]
logofaudit
Comment 13 Peter Krempa 2014-12-01 13:20:59 UTC 
(In reply to Shanzhi Yu from comment #11)
> Created attachment 962397 [details]
> logoflibvirtd

The log doesn't contain section where the snapshot fails. Please attach the correct log.
Comment 14 Shanzhi Yu 2014-12-02 03:12:12 UTC 
Created attachment 963483 [details]
Log of Libvirtd
Comment 15 Shanzhi Yu 2014-12-25 05:31:58 UTC 
I try again with libvirt-1.2.8-11.el7.x86_64, after many times test, I can't reproduce this bug again. 
So change this to VERIFIED status
Comment 17 errata-xmlrpc 2015-03-05 07:46:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html
Note You need to log in before you can comment on or make changes to this bug.