Bug 1002115

Summary: VM fails to start from local storage domain due to: could not open disk image - Permission denied
Product: Red Hat Enterprise Linux 6 Reporter: Gadi Ickowicz <gickowic>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5CC: acathrow, eblake, eedri, gickowic, iheim, italkohe, jdenemar, nlevinki
Target Milestone: rcKeywords: Regression, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-02 08:19:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirt, vdsm + qemu logs none

Description Gadi Ickowicz 2013-08-28 13:26:41 UTC
Created attachment 791399 [details]
libvirt, vdsm + qemu logs

Description of problem:
Attempting to start a start vm with a disk on a local storage domain fails with the following error in vdsm logs:

Thread-274::ERROR::2013-08-28 15:54:12,811::vm::2062::vm.Vm::(_startUnderlyingVm) vmId=`c4a4ced9-d8c2-4cf7-afed-969c70858d87`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 2022, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/vm.py", line 2917, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2662, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error process exited while connecting to monitor: qemu-kvm: -drive file=/rhev/data-center/02f47292-25d7-4058-99ff-e709d3fbab0a/a4474389-b66c-4fc0-b8b4-2a23bdfef4a1/images/35540597-c861-47b8-8acd-99ef71cbc9df/82c1b34e-f349-4e42-8516-d0efc752bbaf,if=none,id=drive-virtio-disk0,format=raw,serial=35540597-c861-47b8-8acd-99ef71cbc9df,cache=none,werror=stop,rerror=stop,aio=threads: could not open disk image /rhev/data-center/02f47292-25d7-4058-99ff-e709d3fbab0a/a4474389-b66c-4fc0-b8b4-2a23bdfef4a1/images/35540597-c861-47b8-8acd-99ef71cbc9df/82c1b34e-f349-4e42-8516-d0efc752bbaf: Permission denied


The file exists:
root@aqua-vds5 qemu]# ll /rhev/data-center/02f47292-25d7-4058-99ff-e709d3fbab0a/a4474389-b66c-4fc0-b8b4-2a23bdfef4a1/images/35540597-c861-47b8-8acd-99ef71cbc9df/82c1b34e-f349-4e42-8516-d0efc752bbaf
-rw-rw----. 1 vdsm kvm 8589934592 Aug 28 15:54 /rhev/data-center/02f47292-25d7-4058-99ff-e709d3fbab0a/a4474389-b66c-4fc0-b8b4-2a23bdfef4a1/images/35540597-c861-47b8-8acd-99ef71cbc9df/82c1b34e-f349-4e42-8516-d0efc752bbaf


however /var/log/messages shows avc denied messages for virsh:
Aug 27 14:22:43 aqua-vds5 kernel: type=1400 audit(1377602563.180:48299): avc:  denied  { create } for  pid=21975 comm="virsh" scontext=unconfined_u:system_r:xm_t:s0 tcontext=unconfined_u:system_r:xm_t:s0 tclass=un
ix_dgram_socket
Aug 27 14:22:43 aqua-vds5 kernel: type=1400 audit(1377602563.182:48300): avc:  denied  { create } for  pid=21975 comm="virsh" scontext=unconfined_u:system_r:xm_t:s0 tcontext=unconfined_u:system_r:xm_t:s0 tclass=un
ix_dgram_socket
Aug 27 14:22:43 aqua-vds5 kernel: type=1400 audit(1377602563.182:48301): avc:  denied  { create } for  pid=21975 comm="virsh" scontext=unconfined_u:system_r:xm_t:s0 tcontext=unconfined_u:system_r:xm_t:s0 tclass=un
ix_dgram_socket


Attempting to create the vm directly from virsh using xml fails with same error:
[root@aqua-vds5 qemu]# virsh create /tmp/vm.xml 
Please enter your authentication name: 1
Please enter your password: 
error: Failed to create domain from /tmp/vm.xml
error: internal error process exited while connecting to monitor: qemu-kvm: -drive file=/rhev/data-center/02f47292-25d7-4058-99ff-e709d3fbab0a/a4474389-b66c-4fc0-b8b4-2a23bdfef4a1/images/35540597-c861-47b8-8acd-99ef71cbc9df/82c1b34e-f349-4e42-8516-d0efc752bbaf,if=none,id=drive-virtio-disk0,format=raw,serial=35540597-c861-47b8-8acd-99ef71cbc9df,cache=none,werror=stop,rerror=stop,aio=threads: could not open disk image /rhev/data-center/02f47292-25d7-4058-99ff-e709d3fbab0a/a4474389-b66c-4fc0-b8b4-2a23bdfef4a1/images/35540597-c861-47b8-8acd-99ef71cbc9df/82c1b34e-f349-4e42-8516-d0efc752bbaf: Permission denied



Version-Release number of selected component (if applicable):
vdsm-4.12.0-72.git287bb7e.el6ev.x86_64
libvirt-0.10.2-22.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.398.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.398.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create local storage DC
2. Create VM with disk
3. Start VM

Actual results:
Fails to start vm

Expected results:
VM should start

Additional info:

Comment 4 Jiri Denemark 2013-08-29 14:03:29 UTC
The AVC denials are completely unrelated to this issues. Since you can reproduce, could you switch SELinux into permissive mode, run the same domain that did not start in enforcing mode, and provide SELinux errors reported in permissive mode?

Comment 5 Jiri Denemark 2013-08-29 14:08:24 UTC
Oh, actually I just realized this could be caused by the incorrect backport of patches for bug 964359. Could you try reproducing this issue with the following scratch build https://brewweb.devel.redhat.com/brew/taskinfo?taskID=6205525 ?

Comment 6 Eric Blake 2013-08-29 14:30:19 UTC
Indeed, these symptoms match those of 964359.  I'd appreciate confirmation from testing with the scratch build, but I'll probably close this as a dup of that one if we don't hear further in a couple days.

Comment 7 Eyal Edri 2013-08-30 15:21:45 UTC
this scratch build no longer contains any rpms - might have been Garbaged collected. 

looking at x86_64 for example:
https://brewweb.devel.redhat.com/taskinfo?taskID=6205528

can you post link to a new build to test?

Comment 8 Eric Blake 2013-08-30 15:29:05 UTC
(In reply to Eyal Edri from comment #7)
> this scratch build no longer contains any rpms - might have been Garbaged
> collected. 

Yep, that happens after a week.  New build is kicked off:

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=6230974

Comment 9 Gadi Ickowicz 2013-09-01 07:11:52 UTC
(In reply to Jiri Denemark from comment #5)
> Oh, actually I just realized this could be caused by the incorrect backport
> of patches for bug 964359. Could you try reproducing this issue with the
> following scratch build
> https://brewweb.devel.redhat.com/brew/taskinfo?taskID=6205525 ?

I tried with the scratch build (http://brewweb.devel.redhat.com/brew/taskinfo?taskID=6230974) and the vm is created successfully

Comment 10 Jiri Denemark 2013-09-02 08:19:37 UTC

*** This bug has been marked as a duplicate of bug 964359 ***