Bug 919074 - Instances fail to boot 'could not open disk image Permission denied'
Summary: Instances fail to boot 'could not open disk image Permission denied'
Keywords:
Status: CLOSED DUPLICATE of bug 915349
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 2.1
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: snapshot5
: 2.1
Assignee: Lon Hohberger
QA Contact: Yaniv Kaul
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-07 14:37 UTC by Dan Prince
Modified: 2019-09-09 13:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-08 21:18:01 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
/var/log/audit/audit.log (2.73 MB, text/plain)
2013-03-07 15:03 UTC, Dan Prince 		no flags Details
View All

Description Dan Prince 2013-03-07 14:37:23 UTC 
As of this morning (March 7th) I'm seeing CI failures when trying to boot an instance. The following exception is getting logged to Nova's compute.log file:

2013-03-07 09:27:14 ERROR nova.compute.manager [req-fb21b35d-ea74-4847-ae95-9e829801eddb db8b786bd8af471abbc4d4e5c6361a85 997b044fe1c64c4b8255b54b884b2eb9] [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c] Instance failed to spawn
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c] Traceback (most recent call last):
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 754, in _spawn
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     block_device_info)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     temp_level, payload)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     self.gen.next()
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     return f(*args, **kw)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1096, in spawn
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     block_device_info)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1961, in _create_domain_and_network
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     domain = self._create_domain(xml)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1940, in _create_domain
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     domain.createWithFlags(launch_flags)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     rv = execute(f,*args,**kwargs)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     rv = meth(*args,**kwargs)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in createWithFlags
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c] libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/1
2013-03-07 09:27:14 11733 TRACE nova.compute.manager [instance: 3a0f08f1-75b7-4c82-a16c-5fb609fb040c] qemu-kvm: -drive file=/var/lib/nova/instances/instance-00000001/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /var/lib/nova/instances/instance-00000001/disk: Permission denied

Version-Release number of selected component (if applicable):

$ rpm -qa | grep openstack
openstack-cinder-2012.2.3-4.el6ost.noarch
openstack-nova-common-2012.2.3-4.el6ost.noarch
openstack-nova-console-2012.2.3-4.el6ost.noarch
openstack-swift-object-1.7.4-9.el6ost.noarch
openstack-selinux-0.1.2-6.el6ost.noarch
openstack-keystone-2012.2.3-4.el6ost.noarch
openstack-glance-2012.2.3-3.el6ost.noarch
openstack-nova-api-2012.2.3-4.el6ost.noarch
openstack-nova-network-2012.2.3-4.el6ost.noarch
openstack-packstack-2012.2.3-0.2.dev454.el6ost.noarch
openstack-nova-scheduler-2012.2.3-4.el6ost.noarch
openstack-nova-compute-2012.2.3-4.el6ost.noarch
openstack-dashboard-2012.2.3-6.el6ost.noarch
openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
openstack-swift-proxy-1.7.4-9.el6ost.noarch
openstack-swift-account-1.7.4-9.el6ost.noarch
openstack-nova-cert-2012.2.3-4.el6ost.noarch
openstack-nova-novncproxy-0.4-3.el6ost.noarch
python-django-openstack-auth-1.0.6-2.el6ost.noarch
openstack-swift-1.7.4-9.el6ost.noarch
openstack-swift-container-1.7.4-9.el6ost.noarch
openstack-utils-2013.1-2.el6ost.noarch


How reproducible:

*Always* (all CI jobs are now consistently failing)

Steps to Reproduce:
1. Run packstack.
2. Upload cirros image to glance.
3. Attempt to boot an instance... it will go into an ERROR state.
Comment 2 Dan Prince 2013-03-07 15:03:03 UTC 
Created attachment 706656 [details]
/var/log/audit/audit.log
Comment 3 Lon Hohberger 2013-03-08 17:38:50 UTC 
type=AVC msg=audit(1362666434.127:9633): avc:  denied  { read } for  pid=12666 comm="qemu-kvm" name="1b2f18e21edadbfb7972bbbbe8d232c5392eb5b9" dev=dm-0 ino=1702204 scontext=unconfined_u:system_r:svirt_t:s0:c436,c850 tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1362666434.127:9633): arch=c000003e syscall=2 success=no exit=-13 a0=7fff7b8a57b0 a1=800 a2=0 a3=0 items=0 ppid=1 pid=12666 auid=0 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=157 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=unconfined_u:system_r:svirt_t:s0:c436,c850 key=(null)
type=AVC msg=audit(1362666434.127:9634): avc:  denied  { getattr } for  pid=12666 comm="qemu-kvm" path="/var/lib/nova/instances/_base/1b2f18e21edadbfb7972bbbbe8d232c5392eb5b9" dev=dm-0 ino=1702204 scontext=unconfined_u:system_r:svirt_t:s0:c436,c850 tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1362666434.127:9634): arch=c000003e syscall=4 success=no exit=-13 a0=7fff7b8a57b0 a1=7fff7b8a3560 a2=7fff7b8a3560 a3=0 items=0 ppid=1 pid=12666 auid=0 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=157 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=unconfined_u:system_r:svirt_t:s0:c436,c850 key=(null)
type=AVC msg=audit(1362666434.127:9635): avc:  denied  { read } for  pid=12666 comm="qemu-kvm" name="1b2f18e21edadbfb7972bbbbe8d232c5392eb5b9" dev=dm-0 ino=1702204 scontext=unconfined_u:system_r:svirt_t:s0:c436,c850 tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1362666434.127:9635): arch=c000003e syscall=2 success=no exit=-13 a0=7fff7b8a57b0 a1=81000 a2=0 a3=40 items=0 ppid=1 pid=12666 auid=0 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=157 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=unconfined_u:system_r:svirt_t:s0:c436,c850 key=(null)

These look a lot like 896013
Comment 10 Lon Hohberger 2013-03-08 21:18:01 UTC 
The patch in 915349 fixes this.

*** This bug has been marked as a duplicate of bug 915349 ***
Note You need to log in before you can comment on or make changes to this bug.