RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/
Bug 1049861 - fail to create snapshot on an "in-use" GlusterFS volume using --force true (el7)
Summary: fail to create snapshot on an "in-use" GlusterFS volume using --force true (el7)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RDO
Classification: Community
Component: openstack-packstack
Version: unspecified
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Martin Magr
QA Contact: yeylon@redhat.com
URL:
Whiteboard: storage
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-08 11:42 UTC by Yogev Rabl
Modified: 2016-04-26 20:19 UTC (History)
6 users (show)

Fixed In Version: openstack-packstack-2014.1.1-0.14.dev1100.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-30 22:59:42 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 91072 0 None None None Never
OpenStack gerrit 96439 0 None None None Never

Description Yogev Rabl 2014-01-08 11:42:11 UTC 
Description of problem:
The create snapshot fails on a fully distributed topology. Cinder fails to create the snapshot due to to an error from the Nova. 

The cinder volume log show:

2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):                                                                                               
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data                        
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     **args)                                                                                                                      
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch                       
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)                                                                             
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 823, in wrapper                                                  
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)                                                                                           
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 381, in create_snapshot                                 
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error'})
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 372, in create_snapshot
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_snapshot(snapshot_ref)
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 428, in create_snapshot
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp     raise exception.GlusterfsException(msg)
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp GlusterfsException: Nova returned "error" status while creating snapshot.
2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp

The nova compute log show:

2014-01-08 13:28:50.617 32429 ERROR nova.openstack.common.rpc.amqp [req-8a59b323-c834-4507-9efc-5a34f9d8043b c2b12ec6592a4259b2e2b45b7f278150 2c398f94ad5e410bb0d8be60ddf60d84] Exception during mess
age handling
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     **args)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 439, in inner
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     return catch_client_exception(exceptions, func, *args, **kwargs)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 420, in catch_client_exception
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     return func(*args, **kwargs)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2479, in volume_snapshot_create
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     create_info)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1755, in volume_snapshot_create
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     context, snapshot_id, 'error')
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1748, in volume_snapshot_create
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     create_info['new_file'])
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1706, in _volume_snapshot_create
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     domain.snapshotCreateXML(snapshot_xml, snap_flags)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp     if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp libvirtError: unsupported configuration: reuse is not supported with this QEMU binary
2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp


Version-Release number of selected component (if applicable):
openstack-nova-common-2014.1-0.5.b1.el6.noarch
python-nova-2014.1-0.5.b1.el6.noarch
openstack-nova-compute-2014.1-0.5.b1.el6.noarch
python-novaclient-2.15.0-1.el6.noarch

openstack-cinder-2014.1-0.2.b1.el6.noarch
python-cinderclient-1.0.7-1.el6.noarch
python-cinder-2014.1-0.2.b1.el6.noarch

How reproducible:
100%

Steps to Reproduce:
1. create a snapshot from an "in-use" volume


Actual results:
fail to create the snapshot

Expected results:
the snapshot should be created successfully.

Additional info:
Comment 1 Yogev Rabl 2014-01-08 11:48:30 UTC 
This bug also effect the instance snapshot action - the same errors appears in the nova compute log.
Comment 2 Eric Harney 2014-01-09 18:34:48 UTC 
This happens because the version of QEMU you are using doesn't support live snapshots.

What version of qemu do you have and how was this installed?
Comment 3 Yogev Rabl 2014-03-17 13:22:27 UTC 
(In reply to Eric Harney from comment #2)
> This happens because the version of QEMU you are using doesn't support live
> snapshots.
> 
> What version of qemu do you have and how was this installed?

it was installed with packstack & the version was  qemu-img-rhev-0.12.1.2-2.415.el6_5.3.x86_64.rpm
Comment 4 Eric Harney 2014-03-26 15:59:05 UTC 
Has this been seen with icehouse-3 packages?  I'm still not sure how to reproduce this.
Comment 5 Yogev Rabl 2014-03-27 08:49:50 UTC 
To reproduce: 
1. Configure Cinder with GlusterFS back end.
2. Create a volume.
3. Attach the volume to an active instance.
4. Try to create a snapshot of that volume: 
# cinder snapshot-create <volume-id> --force True


In addition, I've tried to reproduce the issue in IceHouse, the result is that the creation of the snapshot failed with the error:

2014-03-27 10:46:26.926 17931 ERROR cinder.openstack.common.rpc.amqp [req-b1af6ae8-37ed-4f2b-843d-aacffadc93f8 None] Exception during message handling
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     **args)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 481, in create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error'})
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 472, in create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_snapshot(snapshot_ref)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/glusterfs.py", line 304, in create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     return self._create_snapshot(snapshot)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/glusterfs.py", line 444, in _create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp     raise exception.GlusterfsException(msg)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp GlusterfsException: Nova returned "error" status while creating snapshot.
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp
Comment 6 Eric Harney 2014-03-27 14:05:48 UTC 
(In reply to Yogev Rabl from comment #5)
> In addition, I've tried to reproduce the issue in IceHouse, the result is
> that the creation of the snapshot failed with the error:
> 
> 2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp
> GlusterfsException: Nova returned "error" status while creating snapshot.
> 2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp

The actual failure that occurs is in the nova compute log.
Comment 9 Yogev Rabl 2014-04-02 12:43:11 UTC 
The compute logs:

2014-04-02 15:41:54.457 8178 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: unsupported configuration: reuse is not supported with this QEMU binary
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2537, in volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     create_info)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1812, in volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     context, snapshot_id, 'error')
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1805, in volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     create_info['new_file'])
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1763, in _volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     domain.snapshotCreateXML(snapshot_xml, snap_flags)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 179, in doit
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 139, in proxy_call
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     rv = execute(f,*args,**kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 77, in tworker
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     rv = meth(*args,**kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1778, in snapshotCreateXML
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher     if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher libvirtError: unsupported configuration: reuse is not supported with this QEMU binary
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher 
2014-04-02 15:41:54.459 8178 ERROR oslo.messaging._drivers.common [-] Returning exception unsupported configuration: reuse is not supported with this QEMU binary to caller
2014-04-02 15:41:54.459 8178 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2537, in volume_snapshot_create\n    create_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1812, in volume_snapshot_create\n    context, snapshot_id, \'error\')\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1805, in volume_snapshot_create\n    create_info[\'new_file\'])\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1763, in _volume_snapshot_create\n    domain.snapshotCreateXML(snapshot_xml, snap_flags)\n', '  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 179, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 139, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 77, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1778, in snapshotCreateXML\n    if ret is None:raise libvirtError(\'virDomainSnapshotCreateXML() failed\', dom=self)\n', 'libvirtError: unsupported configuration: reuse is not supported with this QEMU binary\n']
Comment 10 Eric Harney 2014-04-02 13:50:33 UTC 
This still seems to point to an unexpected version of qemu being installed: libvirtError: unsupported configuration: reuse is not supported with this QEMU binary

(Note that before I said qemu-img which is not the correct package to look at.)

Can you post the output of "rpm -qa qemu\*" ?
Comment 11 Yogev Rabl 2014-04-02 13:58:16 UTC 
sure: 
qemu-kvm-1.5.3-59.el7.x86_64
qemu-kvm-common-1.5.3-59.el7.x86_64
qemu-img-1.5.3-59.el7.x86_64
Comment 12 Eric Harney 2014-04-03 19:05:39 UTC 
You need qemu-kvm-rhev and qemu-img-rhev for this to work, the base RHEL versions of KVM do not support the required features.  Reassigning to packstack.

Packstack should not create a system configured this way.
Note You need to log in before you can comment on or make changes to this bug.