This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1023040 - cinder [glusterfs]: cannot attach a volume to an instance running on a compute without cinder installed
cinder [glusterfs]: cannot attach a volume to an instance running on a comput...
Status: CLOSED DUPLICATE of bug 1023065
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
4.0
x86_64 Linux
unspecified Severity high
: ---
: 4.0
Assigned To: RHOS Maint
Haim
storage
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-24 09:40 EDT by Dafna Ron
Modified: 2016-04-26 15:30 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-28 15:52:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (19.35 MB, application/x-gzip)
2013-10-24 09:40 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1244213 None None None Never

  None (edit)
Description Dafna Ron 2013-10-24 09:40:27 EDT
Created attachment 815788 [details]
logs

Description of problem:

I installed a setup with glusterfs as cinder backend and 2 computes (so AIO+extra compute with gluster as storage). 

when I run an instance on the second compute and try to attach a volume to it we fail. 

Version-Release number of selected component (if applicable):

openstack-cinder-2013.2-0.11.rc1.el6ost.noarch
qemu-img-rhev-0.12.1.2-2.413.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.413.el6.x86_64
fuse-2.8.3-4.el6.x86_64
fuse-libs-2.8.3-4.el6.x86_64
glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64

How reproducible:

100%

Steps to Reproduce:
1. install openstack with AIO+a compute with glusterfs using packstack
2. create volumes 
3. run instances on second compute
4. try to attach volume to the compute 

Actual results:

we fail 

Expected results:

we should succeed

Additional info: not sure is this is a cinder issue or a config issue that should be sorted by packstack during install. 

compute log from second compute: 

2013-10-24 15:56:12.849 28700 DEBUG qpid.messaging [-] RETR[38313b0]: Message(properties={'x-amqp-0-10.routing-key': u'reply_68df19f254a74fcb9c38e31e1ae8454d'}, content={u'oslo.message': u'{"_unique_id": "1912451198484b169b1be38a8380de7c", "failure": null, "_msg_id": "6cad2d4d13e24f3cade178cc5950e688", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654
2013-10-24 15:56:12.849 28700 DEBUG qpid.messaging.io.ops [-] SENT[37a3710]: MessageFlow(destination='0', unit=0, value=1L, id=serial(32082)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-24 15:56:12.850 28700 DEBUG qpid.messaging.io.ops [-] SENT[37a3710]: SessionCompleted(commands=[0-32075]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-24 15:56:12.855 28700 ERROR nova.openstack.common.rpc.amqp [req-adfcf895-7067-41c6-9137-48604ac5e619 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Exception during message handling
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     **args)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     payload)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     pass
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3638, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     context, instance, mountpoint)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3633, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     mountpoint, instance)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3679, in _attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     connector)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3669, in _attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     encryption=encryption)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1105, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     disk_dev)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1092, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     virt_dom.attachDeviceFlags(conf.to_xml(), flags)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 419, in attachDeviceFlags
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk2' could not be initialized
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp 
2013-10-24 15:56:12.858 28700 DEBUG qpid.messaging.io.raw [-] SENT[37a3710]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00}K' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480

volumes log shows rootwrap error but I do not think that its connected to this issue since times do not fit. 

2013-10-24 15:26:46.747 2483 ERROR cinder.openstack.common.rpc.common [req-09b4b078-346e-40f4-89fe-08e50ced7173 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data\n    **args)\n', '  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch\n    return getattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper\n    return func(self, *args, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 605, in initialize_connection\n    conn_info = self.driver.initialize_connection(volume, connector)\n', '  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 851, in initialize_connection\n    info = self._qemu_img_info(path)\n', '  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 132, in _qemu_img_info\n    info = image_utils.qemu_img_info(path)\n', '  File "/usr/lib/python2.6/site-packages/cinder/image/image_utils.py", line 191, in qemu_img_info\n    out, err = utils.execute(*cmd, run_as_root=True)\n', '  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute\n    return processutils.execute(*cmd, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute\n    cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452\nExit code: 1\nStdout: \'\'\nStderr: "Could not open \'/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452\': No such file or directory\\n"\n']
2013-10-24 15:26:51.034 2483 ERROR cinder.openstack.common.rpc.amqp [req-31326b3f-cee3-4600-b3a3-f784071c8e19 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Exception during message handling
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 605, in initialize_connection
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     conn_info = self.driver.initialize_connection(volume, connector)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 851, in initialize_connection
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     info = self._qemu_img_info(path)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 132, in _qemu_img_info
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     info = image_utils.qemu_img_info(path)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/image/image_utils.py", line 191, in qemu_img_info
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     out, err = utils.execute(*cmd, run_as_root=True)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Stdout: ''
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Stderr: "Could not open '/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452': No such file or directory\n"
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp 
2013-10-24 15:26:51.037 2483 ERROR cinder.openstack.common.rpc.common [req-31326b3f-cee3-4600-b3a3-f784071c8e19 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Returning exception Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452
Exit code: 1
Stdout: ''
Stderr: "Could not open '/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452': No such file or directory\n" to caller
Comment 1 Dafna Ron 2013-10-24 09:46:41 EDT
further tests I am confirming that the rootwrap is caused when trying to boot a cloned volume and is not related to this issue.
Comment 3 Eric Harney 2013-10-28 15:52:00 EDT

*** This bug has been marked as a duplicate of bug 1023065 ***

Note You need to log in before you can comment on or make changes to this bug.