Bug 1051661 - GlusterFS driver synchronization for concurrent operations
Summary: GlusterFS driver synchronization for concurrent operations
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 4.0
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: z2
: 4.0
Assignee: Eric Harney
QA Contact: Dafna Ron
URL:
Whiteboard:
Depends On:
Blocks: 984280 1045196
TreeView+ depends on / blocked
 
Reported: 2014-01-10 19:54 UTC by Eric Harney
Modified: 2016-04-26 16:16 UTC (History)
4 users (show)

Fixed In Version: openstack-cinder-2013.2.1-7.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, GlusterFS clone from volume used the same temporary snapshot filename for concurrent clone operations from the same volume. As a result, one of the clone operations would fail. With this update, a locking feature is added. As a result, even if multiple volume clone operations are issued at the same time, they will happen sequentially rather than failing.
Clone Of:
Environment:
Last Closed: 2014-03-04 20:13:57 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 66047 0 None None None Never
OpenStack gerrit 67898 0 None None None Never
OpenStack gerrit 68789 0 None None None Never
Red Hat Product Errata RHBA-2014:0213 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform 4 Bug Fix and Enhancement Advisory 2014-03-05 01:11:55 UTC

Description Eric Harney 2014-01-10 19:54:44 UTC 
Some GlusterFS operations are still not synchronized/locked, which causes failures like the below (multiple clones from the same volume running at once, reported in bug 984280):


2013-12-11 19:39:54.425 3860 TRACE cinder.volume.flows.create_volume Stderr: 'Error: Trying to create an image with the same filename as the backing file\n'
2013-12-11 19:39:54.425 3860 TRACE cinder.volume.flows.create_volume 
2013-12-11 19:39:54.428 3860 ERROR cinder.openstack.common.rpc.amqp [req-517c7870-adc9-4cfd-a50f-eb0a9a03ac12 674776fc1eea47718301aeacbab072b3 5cea8d9e58c841dfb03b1cda755b539d] Exception during message handling
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 809, in wrapper
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     flow.run(context.elevated())
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     return f(self, *args, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     run_it(r)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     self.rollback(context, cause)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     result = runner(context, *args, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     self.result = self.task(*args, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     **volume_spec)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source_volume
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 188, in create_cloned_volume
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     self.create_snapshot(temp_snapshot)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 460, in create_snapshot
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     self._create_snapshot(snapshot, path_to_disk, snap_id)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 500, in _create_snapshot
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     new_snap_path)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 478, in _create_qcow2_snap_file
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     self._execute(*command, run_as_root=True)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 143, in execute
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/249458a2755cd0a9f302b9d81eb3f35d/volume-c5bfe327-59f2-4936-846b-a0a88b8c6687.tmp-snap-c5bfe327-59f2-4936-846b-a0a88b8c6687 /var/lib/cinder/mnt/249458a2755cd0a9f302b9d81eb3f35d/volume-c5bfe327-59f2-4936-846b-a0a88b8c6687.tmp-snap-c5bfe327-59f2-4936-846b-a0a88b8c6687
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp Stdout: ''
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp Stderr: 'Error: Trying to create an image with the same filename as the backing file\n'
2013-12-11 19:39:54.428 3860 TRACE cinder.openstack.common.rpc.amqp 
(END)
Comment 4 Eric Harney 2014-01-23 16:43:43 UTC 
Found a deadlock issue while testing this patch.  create_snapshot tries to acquire the same lock that create_cloned_volume is holding.
Comment 6 Dafna Ron 2014-02-18 16:55:12 UTC 
verified on openstack-cinder-2013.2.2-1.el6ost.noarch
Comment 7 Dafna Ron 2014-02-18 16:55:28 UTC 
verified on openstack-cinder-2013.2.2-1.el6ost.noarch
Comment 9 errata-xmlrpc 2014-03-04 20:13:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0213.html
Note You need to log in before you can comment on or make changes to this bug.