Bug 1049861
| Summary: | fail to create snapshot on an "in-use" GlusterFS volume using --force true (el7) | ||
|---|---|---|---|
| Product: | [Community] RDO | Reporter: | Yogev Rabl <yrabl> |
| Component: | openstack-packstack | Assignee: | Martin Magr <mmagr> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | yeylon <yeylon> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | unspecified | CC: | aortega, derekh, mmagr, srevivo, yeylon, yrabl |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | storage | ||
| Fixed In Version: | openstack-packstack-2014.1.1-0.14.dev1100.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-03-30 22:59:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This bug also effect the instance snapshot action - the same errors appears in the nova compute log. This happens because the version of QEMU you are using doesn't support live snapshots. What version of qemu do you have and how was this installed? (In reply to Eric Harney from comment #2) > This happens because the version of QEMU you are using doesn't support live > snapshots. > > What version of qemu do you have and how was this installed? it was installed with packstack & the version was qemu-img-rhev-0.12.1.2-2.415.el6_5.3.x86_64.rpm Has this been seen with icehouse-3 packages? I'm still not sure how to reproduce this. To reproduce:
1. Configure Cinder with GlusterFS back end.
2. Create a volume.
3. Attach the volume to an active instance.
4. Try to create a snapshot of that volume:
# cinder snapshot-create <volume-id> --force True
In addition, I've tried to reproduce the issue in IceHouse, the result is that the creation of the snapshot failed with the error:
2014-03-27 10:46:26.926 17931 ERROR cinder.openstack.common.rpc.amqp [req-b1af6ae8-37ed-4f2b-843d-aacffadc93f8 None] Exception during message handling
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp **args)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 481, in create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp {'status': 'error'})
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 472, in create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_snapshot(snapshot_ref)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/glusterfs.py", line 304, in create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp return self._create_snapshot(snapshot)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/glusterfs.py", line 444, in _create_snapshot
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp raise exception.GlusterfsException(msg)
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp GlusterfsException: Nova returned "error" status while creating snapshot.
2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp
(In reply to Yogev Rabl from comment #5) > In addition, I've tried to reproduce the issue in IceHouse, the result is > that the creation of the snapshot failed with the error: > > 2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp > GlusterfsException: Nova returned "error" status while creating snapshot. > 2014-03-27 10:46:26.926 17931 TRACE cinder.openstack.common.rpc.amqp The actual failure that occurs is in the nova compute log. The compute logs:
2014-04-02 15:41:54.457 8178 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: unsupported configuration: reuse is not supported with this QEMU binary
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2537, in volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher create_info)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1812, in volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher context, snapshot_id, 'error')
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1805, in volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher create_info['new_file'])
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1763, in _volume_snapshot_create
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher domain.snapshotCreateXML(snapshot_xml, snap_flags)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 179, in doit
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 139, in proxy_call
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher rv = execute(f,*args,**kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 77, in tworker
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher rv = meth(*args,**kwargs)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1778, in snapshotCreateXML
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher libvirtError: unsupported configuration: reuse is not supported with this QEMU binary
2014-04-02 15:41:54.457 8178 TRACE oslo.messaging.rpc.dispatcher
2014-04-02 15:41:54.459 8178 ERROR oslo.messaging._drivers.common [-] Returning exception unsupported configuration: reuse is not supported with this QEMU binary to caller
2014-04-02 15:41:54.459 8178 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n return func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2537, in volume_snapshot_create\n create_info)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1812, in volume_snapshot_create\n context, snapshot_id, \'error\')\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1805, in volume_snapshot_create\n create_info[\'new_file\'])\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1763, in _volume_snapshot_create\n domain.snapshotCreateXML(snapshot_xml, snap_flags)\n', ' File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 179, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 139, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 77, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1778, in snapshotCreateXML\n if ret is None:raise libvirtError(\'virDomainSnapshotCreateXML() failed\', dom=self)\n', 'libvirtError: unsupported configuration: reuse is not supported with this QEMU binary\n']
This still seems to point to an unexpected version of qemu being installed: libvirtError: unsupported configuration: reuse is not supported with this QEMU binary (Note that before I said qemu-img which is not the correct package to look at.) Can you post the output of "rpm -qa qemu\*" ? sure: qemu-kvm-1.5.3-59.el7.x86_64 qemu-kvm-common-1.5.3-59.el7.x86_64 qemu-img-1.5.3-59.el7.x86_64 You need qemu-kvm-rhev and qemu-img-rhev for this to work, the base RHEL versions of KVM do not support the required features. Reassigning to packstack. Packstack should not create a system configured this way. |
Description of problem: The create snapshot fails on a fully distributed topology. Cinder fails to create the snapshot due to to an error from the Nova. The cinder volume log show: 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp **args) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 823, in wrapper 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 381, in create_snapshot 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp {'status': 'error'}) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp self.gen.next() 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 372, in create_snapshot 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_snapshot(snapshot_ref) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 428, in create_snapshot 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp raise exception.GlusterfsException(msg) 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp GlusterfsException: Nova returned "error" status while creating snapshot. 2014-01-08 13:28:51.665 7705 TRACE cinder.openstack.common.rpc.amqp The nova compute log show: 2014-01-08 13:28:50.617 32429 ERROR nova.openstack.common.rpc.amqp [req-8a59b323-c834-4507-9efc-5a34f9d8043b c2b12ec6592a4259b2e2b45b7f278150 2c398f94ad5e410bb0d8be60ddf60d84] Exception during mess age handling 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp **args) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 439, in inner 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp return catch_client_exception(exceptions, func, *args, **kwargs) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 420, in catch_client_exception 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp return func(*args, **kwargs) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2479, in volume_snapshot_create 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp create_info) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1755, in volume_snapshot_create 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp context, snapshot_id, 'error') 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1748, in volume_snapshot_create 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp create_info['new_file']) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1706, in _volume_snapshot_create 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp domain.snapshotCreateXML(snapshot_xml, snap_flags) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwargs) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp rv = execute(f,*args,**kwargs) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp rv = meth(*args,**kwargs) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self) 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp libvirtError: unsupported configuration: reuse is not supported with this QEMU binary 2014-01-08 13:28:50.617 32429 TRACE nova.openstack.common.rpc.amqp Version-Release number of selected component (if applicable): openstack-nova-common-2014.1-0.5.b1.el6.noarch python-nova-2014.1-0.5.b1.el6.noarch openstack-nova-compute-2014.1-0.5.b1.el6.noarch python-novaclient-2.15.0-1.el6.noarch openstack-cinder-2014.1-0.2.b1.el6.noarch python-cinderclient-1.0.7-1.el6.noarch python-cinder-2014.1-0.2.b1.el6.noarch How reproducible: 100% Steps to Reproduce: 1. create a snapshot from an "in-use" volume Actual results: fail to create the snapshot Expected results: the snapshot should be created successfully. Additional info: