Created attachment 815788 [details]
logs

Description of problem:

I installed a setup with glusterfs as cinder backend and 2 computes (so AIO+extra compute with gluster as storage). 

when I run an instance on the second compute and try to attach a volume to it we fail. 

Version-Release number of selected component (if applicable):

openstack-cinder-2013.2-0.11.rc1.el6ost.noarch
qemu-img-rhev-0.12.1.2-2.413.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.413.el6.x86_64
fuse-2.8.3-4.el6.x86_64
fuse-libs-2.8.3-4.el6.x86_64
glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64

How reproducible:

100%

Steps to Reproduce:
1. install openstack with AIO+a compute with glusterfs using packstack
2. create volumes 
3. run instances on second compute
4. try to attach volume to the compute 

Actual results:

we fail 

Expected results:

we should succeed

Additional info: not sure is this is a cinder issue or a config issue that should be sorted by packstack during install. 

compute log from second compute: 

2013-10-24 15:56:12.849 28700 DEBUG qpid.messaging [-] RETR[38313b0]: Message(properties={'x-amqp-0-10.routing-key': u'reply_68df19f254a74fcb9c38e31e1ae8454d'}, content={u'oslo.message': u'{"_unique_id": "1912451198484b169b1be38a8380de7c", "failure": null, "_msg_id": "6cad2d4d13e24f3cade178cc5950e688", "result": null, "ending": true}', u'oslo.version': u'2.0'}) _get /usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py:654
2013-10-24 15:56:12.849 28700 DEBUG qpid.messaging.io.ops [-] SENT[37a3710]: MessageFlow(destination='0', unit=0, value=1L, id=serial(32082)) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-24 15:56:12.850 28700 DEBUG qpid.messaging.io.ops [-] SENT[37a3710]: SessionCompleted(commands=[0-32075]) write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:686
2013-10-24 15:56:12.855 28700 ERROR nova.openstack.common.rpc.amqp [req-adfcf895-7067-41c6-9137-48604ac5e619 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Exception during message handling
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     **args)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     payload)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     pass
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258, in decorated_function
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3638, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     context, instance, mountpoint)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3633, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     mountpoint, instance)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3679, in _attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     connector)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3669, in _attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     encryption=encryption)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1105, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     disk_dev)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1092, in attach_volume
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     virt_dom.attachDeviceFlags(conf.to_xml(), flags)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 419, in attachDeviceFlags
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk2' could not be initialized
2013-10-24 15:56:12.855 28700 TRACE nova.openstack.common.rpc.amqp 
2013-10-24 15:56:12.858 28700 DEBUG qpid.messaging.io.raw [-] SENT[37a3710]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00}K' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480

volumes log shows rootwrap error but I do not think that its connected to this issue since times do not fit. 

2013-10-24 15:26:46.747 2483 ERROR cinder.openstack.common.rpc.common [req-09b4b078-346e-40f4-89fe-08e50ced7173 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data\n    **args)\n', '  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch\n    return getattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper\n    return func(self, *args, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 605, in initialize_connection\n    conn_info = self.driver.initialize_connection(volume, connector)\n', '  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 851, in initialize_connection\n    info = self._qemu_img_info(path)\n', '  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 132, in _qemu_img_info\n    info = image_utils.qemu_img_info(path)\n', '  File "/usr/lib/python2.6/site-packages/cinder/image/image_utils.py", line 191, in qemu_img_info\n    out, err = utils.execute(*cmd, run_as_root=True)\n', '  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute\n    return processutils.execute(*cmd, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute\n    cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452\nExit code: 1\nStdout: \'\'\nStderr: "Could not open \'/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452\': No such file or directory\\n"\n']
2013-10-24 15:26:51.034 2483 ERROR cinder.openstack.common.rpc.amqp [req-31326b3f-cee3-4600-b3a3-f784071c8e19 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Exception during message handling
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 605, in initialize_connection
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     conn_info = self.driver.initialize_connection(volume, connector)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 851, in initialize_connection
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     info = self._qemu_img_info(path)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 132, in _qemu_img_info
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     info = image_utils.qemu_img_info(path)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/image/image_utils.py", line 191, in qemu_img_info
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     out, err = utils.execute(*cmd, run_as_root=True)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Stdout: ''
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp Stderr: "Could not open '/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452': No such file or directory\n"
2013-10-24 15:26:51.034 2483 TRACE cinder.openstack.common.rpc.amqp 
2013-10-24 15:26:51.037 2483 ERROR cinder.openstack.common.rpc.common [req-31326b3f-cee3-4600-b3a3-f784071c8e19 a660044c9b074450aaa45fba0d641fcc e27aae2598b94dca88cd0408406e0848] Returning exception Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C LANG=C qemu-img info /var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452
Exit code: 1
Stdout: ''
Stderr: "Could not open '/var/lib/cinder/mnt/792e7ed79ec67a83b6a55e1479a7c82f/volume-0466799b-0810-4c69-a894-0f395fe89452': No such file or directory\n" to caller