| Summary: | cinder [glusterfs]: cannot attach a volume to an instance running on a compute without cinder installed | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Dafna Ron <dron> | ||||
| Component: | openstack-cinder | Assignee: | RHOS Maint <rhos-maint> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Haim <hateya> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.0 | CC: | eharney, fpercoco, hateya, yeylon | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | storage | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2013-10-28 19:52:00 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
further tests I am confirming that the rootwrap is caused when trying to boot a cloned volume and is not related to this issue. *** This bug has been marked as a duplicate of bug 1023065 *** |
Created attachment 815788 [details] logs Description of problem: I installed a setup with glusterfs as cinder backend and 2 computes (so AIO+extra compute with gluster as storage). when I run an instance on the second compute and try to attach a volume to it we fail. Version-Release number of selected component (if applicable): openstack-cinder-2013.2-0.11.rc1.el6ost.noarch qemu-img-rhev-0.12.1.2-2.413.el6.x86_64 qemu-kvm-rhev-0.12.1.2-2.413.el6.x86_64 fuse-2.8.3-4.el6.x86_64 fuse-libs-2.8.3-4.el6.x86_64 glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64 How reproducible: 100% Steps to Reproduce: 1. install openstack with AIO+a compute with glusterfs using packstack 2. create volumes 3. run instances on second compute 4. try to attach volume to the compute Actual results: we fail Expected results: we should succeed Additional info: not sure is this is a cinder issue or a config issue that should be sorted by packstack during install. compute log from second compute: 2013-10-24 15:56:12.849 28700 DEBUG qpid.messaging [-] RETR[38313b0]: Message(properties={'x-amqp-0-10.routing-key': u'reply_68df19f254a74fcb9c38e31e1ae8454d'}, content={u'oslo.message': u'{"_unique_id": "1912451198484b169b1be38a8380de7c", "failure": null, "_msg_id": "6cad2d4d13e24f3cade178cc5950e688", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654 2013-10-24 15:56:12.849 28700 DEBUG qpid.messaging.io.ops [-] SENT[37a3710]: MessageFlow(destination='0', unit=0, value=1L, id=serial(32082)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686 2013-10-24 15:56:12.850 28700 DEBUG qpid.messaging.io.ops [-] SENT[37a3710]: SessionCompleted(commands=[0-32075]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686 2013-10-24 15:56:12.855 28700 ERROR nova.openstack.common.rpc.amqp [req-adfcf895-7067-41c6-9137-48604ac5e619 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Exception during message handling 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp **args) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp payload) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp pass 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3638, in attach_volume 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3633, in attach_volume 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp mountpoint, instance) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3679, in _attach_volume 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp connector) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3669, in _attach_volume 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp encryption=encryption) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1105, in attach_volume 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp disk_dev) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1092, in attach_volume 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp virt_dom.attachDeviceFlags(conf.to_xml(), flags) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwargs) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp rv = execute(f,*args,**kwargs) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp rv = meth(*args,**kwargs) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/site-packages/libvirt.py", line 419, in attachDeviceFlags 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk2' could not be initialized 2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp 2013-10-24 15:56:12.858 28700 DEBUG qpid.messaging.io.raw [-] SENT[37a3710]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00}K' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480 volumes log shows rootwrap error but I do not think that its connected to this issue since times do not fit. 2013-10-24 15:26:46.747 2483 ERROR cinder.openstack.common.rpc.common [req-09b4b078-346e-40f4-89fe-08e50ced7173 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data\n **args)\n', ' File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch\n return getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper\n return func(self, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 605, in initialize_connection\n conn_info = self.driver.initialize_connection(volume, connector)\n', ' File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 851, in initialize_connection\n info = self._qemu_img_info(path)\n', ' File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 132, in _qemu_img_info\n info = image_utils.qemu_img_info(path)\n', ' File "/usr/lib/python2.6/site-packages/cinder/image/image_utils.py", line 191, in qemu_img_info\n out, err = utils.execute(*cmd, run_as_root=True)\n', ' File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute\n cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452\nExit code: 1\nStdout: \'\'\nStderr: "Could not open \'/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452\': No such file or directory\\n"\n'] 2013-10-24 15:26:51.034 2483 ERROR cinder.openstack.common.rpc.amqp [req-31326b3f-cee3-4600-b3a3-f784071c8e19 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Exception during message handling 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp **args) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 605, in initialize_connection 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp conn_info = self.driver.initialize_connection(volume, connector) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 851, in initialize_connection 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp info = self._qemu_img_info(path) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 132, in _qemu_img_info 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp info = image_utils.qemu_img_info(path) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/image/image_utils.py", line 191, in qemu_img_info 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp out, err = utils.execute(*cmd, run_as_root=True) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp return processutils.execute(*cmd, **kwargs) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp cmd=' '.join(cmd)) 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command. 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Exit code: 1 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Stdout: '' 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Stderr: "Could not open '/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452': No such file or directory\n" 2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp 2013-10-24 15:26:51.037 2483 ERROR cinder.openstack.common.rpc.common [req-31326b3f-cee3-4600-b3a3-f784071c8e19 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Returning exception Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452 Exit code: 1 Stdout: '' Stderr: "Could not open '/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452': No such file or directory\n" to caller