| Summary: | GlusterFS: can't create a second snapshot (online) after deleting the first one (online) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Dafna Ron <dron> | ||||||
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Yogev Rabl <yrabl> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 4.0 | CC: | abaron, breeler, dron, eharney, jhenner, mlopes, sgordon, smanjara, yeylon | ||||||
| Target Milestone: | z1 | Keywords: | ZStream | ||||||
| Target Release: | 4.0 | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | storage | ||||||||
| Fixed In Version: | openstack-cinder-2013.2.1-2.el6ost | Doc Type: | Bug Fix | ||||||
| Doc Text: |
Previously, deleting the oldest Block Storage GlusterFS snapshot for attached volumes would leave the volume/snapshot data in an inconsistent state.
This has been fixed, so now deleting the oldest Block Storage GlusterFS snapshot for attached volumes is done correctly.
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2014-01-23 14:22:59 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Bug Depends On: | 1016798, 1040711, 1056037 | ||||||||
| Bug Blocks: | 1033714, 1045196 | ||||||||
| Attachments: |
|
||||||||
It is likely that this bug is being / will be fixed via some of the other bugs in this area related to permissions. Did any other errors appear in the log at snapshot creation/deletion time for the first snapshot? (In reply to Eric Harney from comment #1) > It is likely that this bug is being / will be fixed via some of the other > bugs in this area related to permissions. > > Did any other errors appear in the log at snapshot creation/deletion time > for the first snapshot? I attached everything. nothing more except what I added in this bug. (In reply to Eric Harney from comment #1) > It is likely that this bug is being / will be fixed via some of the other > bugs in this area related to permissions. > > Did any other errors appear in the log at snapshot creation/deletion time > for the first snapshot? Please block on relevant bugs (In reply to Ayal Baron from comment #3) > (In reply to Eric Harney from comment #1) > > It is likely that this bug is being / will be fixed via some of the other > > bugs in this area related to permissions. > > > > Did any other errors appear in the log at snapshot creation/deletion time > > for the first snapshot? > > Please block on relevant bugs I attached relevant logs when I opened the bug. This will no longer happen with 1016798 fixed. (Same issue.) the create succeeds but delete of second snapshot will fail with what seems to me as same error.
moving back to dev.
reproduce:
1. create a volume
2. boot instance from the volume
3. create a snapshot (cinder snapshot-create <vol-id> --force True --display-name test)
4. delete the snapshot (cinder snapshot-delete <snapshot>)
5. create a second snapshot (cinder snapshot-create <vol-id> --force True --display-name test1)
6. delete the second snapshot (cinder snapshot-delete <snapshot>)
2013-12-11 14:48:45.636 3860 WARNING cinder.quota [req-aae0a49b-bd70-4074-be3e-50b5ddee2a3a 674776fc1eea47718301aeacbab072b3 5cea8d9e58c841dfb03b1cda755b539d] Deprecated: Default quota for resource: gigabytes is set by the default quota flag: quota_gigabytes, it is now deprecated. Please use the the default quota class for default quota.
2013-12-11 14:48:45.636 3860 WARNING cinder.quota [req-aae0a49b-bd70-4074-be3e-50b5ddee2a3a 674776fc1eea47718301aeacbab072b3 5cea8d9e58c841dfb03b1cda755b539d] Deprecated: Default quota for resource: snapshots is set by the default quota flag: quota_snapshots, it is now deprecated. Please use the the default quota class for default quota.
2013-12-11 14:48:56.176 3860 ERROR cinder.openstack.common.rpc.amqp [req-c2a0a7ed-45bc-4065-b013-a43b9b37dc3d 674776fc1eea47718301aeacbab072b3 5cea8d9e58c841dfb03b1cda755b539d] Exception during message handling
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp **args)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 809, in wrapper
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 371, in create_snapshot
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp {'status': 'error'})
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 362, in create_snapshot
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_snapshot(snapshot_ref)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 389, in create_snapshot
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp new_snap_path)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 478, in _create_qcow2_snap_file
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp self._execute(*command, run_as_root=True)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 143, in execute
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp return processutils.execute(*cmd, **kwargs)
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp cmd=' '.join(cmd))
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/249458a2755cd0a9f302b9d81eb3f35d/volume-8e113c6f-f30c-4f6a-b511-2af507d7e758.8820d941-904e-49ba-8582-453b533cd033 /var/lib/cinder/mnt/249458a2755cd0a9f302b9d81eb3f35d/volume-8e113c6f-f30c-4f6a-b511-2af507d7e758.1264bdc0-fc0a-4817-976c-ce061a787d92
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp Exit code: 1
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp Stdout: ''
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp Stderr: "Could not open '/var/lib/cinder/mnt/249458a2755cd0a9f302b9d81eb3f35d/volume-8e113c6f-f30c-4f6a-b511-2af507d7e758.8820d941-904e-49ba-8582-453b533cd033': No such file or directory\n"
2013-12-11 14:48:56.176 3860 TRACE cinder.openstack.common.rpc.amqp
Created attachment 835273 [details]
logs
Please provide "Doc Text" for this bug describing the issue and suggested workaround. Problem appears to be an issue with how we call libvirt from Nova to perform a blockRebase, used when deleting the last snapshot. We pass base=<filename> but should pass base=None. The current method appears to rebase but leaves the qcow2 file with an incorrect qcow2 pointer to a file that doesn't exist. Note: Bug 1040711 still exists and affects GlusterFS snapshot deletion for attached volumes, but should be less severe. I think this fix is still worth verifying independently. At the moment I'm not able to verify the bug, because it is blocked by Bug 1052969 the bug that blocks this bug has a target release in A2. Please change the one of the milestones. (In reply to Yogev Rabl from comment #19) Done I tested this bug with 1052969
we either have a new bug or this is not solved:
We fail to delete the snapshot:
2014-01-17 17:12:30.036 9009 ERROR cinder.openstack.common.rpc.amqp [req-2772b058-94f1-4d15-a7d1-661b0682dfed e6ee6034307247b78807be047bd10e76 d4aaa7c237054d408a65f40bb4ee74d0] Exception during message handling
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp **args)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 820, in wrapper
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 424, in delete_snapshot
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'})
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 412, in delete_snapshot
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp self.driver.delete_snapshot(snapshot_ref)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 247, in inner
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 636, in delete_snapshot
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp online_delete_info)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 801, in _delete_snapshot_online
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp raise exception.GlusterfsException(msg)
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp GlusterfsException: Unable to delete snapshot ade7e0e8-e8f0-44da-9308-b41aaed609ac, status: error_deleting.
2014-01-17 17:12:30.036 9009 TRACE cinder.openstack.common.rpc.amqp
/var/log/cinder/volume.log (END)
2014-01-17 17:12:29.138 15658 ERROR nova.openstack.common.rpc.amqp [req-1fae1101-f20a-424e-9ea8-74367d0faf88 e6ee6034307247b78807be047bd10e76 d4aaa7c237054d408a65f40bb4ee74d0] Exception during message handling
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp **args)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 439, in inner
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp return catch_client_exception(exceptions, func, *args, **kwargs)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 420, in catch_client_exception
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp return func(*args, **kwargs)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2365, in volume_snapshot_delete
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp snapshot_id, delete_info)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1808, in volume_snapshot_delete
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp context, snapshot_id, 'error_deleting')
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1801, in volume_snapshot_delete
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp snapshot_id, delete_info=delete_info)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1773, in _volume_snapshot_delete
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp abort_on_error=True):
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1442, in _wait_for_block_job
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp status = domain.blockJobInfo(disk_path, 0)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp rv = execute(f,*args,**kwargs)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp rv = meth(*args,**kwargs)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1826, in blockJobInfo
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp if ret is None: raise libvirtError ('virDomainGetBlockJobInfo() failed', dom=self)
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp libvirtError: virDomainGetBlockJobInfo() failed
2014-01-17 17:12:29.138 15658 TRACE nova.openstack.common.rpc.amqp
2014-01-17 17:12:32.411 15658 DEBUG qpid.messaging.io.ops [-] SENT[3aea908]: ConnectionHeartbeat() write_op /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:685
2014-01-17 17:12:32.413 15658 DEBUG qpid.messaging.io.raw [-] SENT[3aea908]: '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480
2014-01-17 17:12:32.417 15658 DEBUG qpid.messaging.io.raw [-] READ[3aea908]: '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' readable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:416
2014-01-17 17:12:32.417 15658 DEBUG qpid.messaging.io.ops [-] RCVD[3aea908]: ConnectionHeartbeat() write /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:653
2014-01-17 17:12:32.774 15658 DEBUG nova.openstack.common.rpc.amqp [-] Making synchronous call on conductor ... multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:553
2014-01-17 17:12:32.775 15658 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is b4947f51e65848ce8b3c57b26865a1f4 multicall /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:556
2014-01-17 17:12:32.775 15658 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is 5c4a6cf79917497aae384ea8e2f95c8d. _add_unique_id /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py:341
2014-01-17 17:12:32.777 15658 DEBUG qpid.messaging.io.ops [-] SENT[37bc638]: MessageTransfer(destination='amq.topic', id=serial(0), sync=True, headers=(DeliveryProperties(routing_key='topic/nova/conductor'
2014-01-17 14:56:10.783+0000: 14991: info : libvirt version: 0.10.2, package: 29.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2013-10-09-06:25:35, x86-026.build.eng.bos.redhat.com)
2014-01-17 14:56:10.783+0000: 14991: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet1" not in key map
2014-01-17 14:56:10.806+0000: 14991: error : virNetDevGetIndex:653 : Unable to get index for interface vnet1: No such device
2014-01-17 14:56:11.842+0000: 14983: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet5" not in key map
2014-01-17 14:56:11.854+0000: 14983: error : virNetDevGetIndex:653 : Unable to get index for interface vnet5: No such device
2014-01-17 14:56:13.260+0000: 14986: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet3" not in key map
2014-01-17 14:56:13.287+0000: 14986: error : virNetDevGetIndex:653 : Unable to get index for interface vnet3: No such device
2014-01-17 14:56:15.909+0000: 14991: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet4" not in key map
2014-01-17 14:56:15.926+0000: 14991: error : virNetDevGetIndex:653 : Unable to get index for interface vnet4: No such device
2014-01-17 14:56:18.128+0000: 14991: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet0" not in key map
2014-01-17 14:56:18.141+0000: 14991: error : virNetDevGetIndex:653 : Unable to get index for interface vnet0: No such device
2014-01-17 14:56:20.379+0000: 14986: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet2" not in key map
2014-01-17 14:56:20.381+0000: 14986: error : virNetDevGetIndex:653 : Unable to get index for interface vnet2: No such device
2014-01-17 15:02:42.376+0000: 14983: error : qemuDomainSnapshotFSFreeze:11112 : argument unsupported: QEMU guest agent is not configured
libvirtError: virDomainGetBlockJobInfo() failed This comes from not having a sufficient version of libvirt for bug 1038815, need libvirt-0.10.2-29.el6_5.2. Your box has libvirt-0.10.2-29.el6.x86_64. delete of snapshot when it's the only snapshot for a volume fails. after discussion with devel, this bug was verified using a volume with more than one snapshot: 1. create a volume 2. boot instance from the volume 3. create two snapshots 4. delete one. the delete succeeded. moving to verified on openstack-cinder-2013.2.1-5.el6ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-0046.html *** Bug 1056469 has been marked as a duplicate of this bug. *** |
Created attachment 827821 [details] logs Description of problem: we get an error when we try to create a second snapshot from a volume which is used as a bootable disk after deleting the first snapshot created Version-Release number of selected component (if applicable): openstack-cinder-2013.2-2.el6ost.noarch How reproducible: 100% scenario #1: 1. create a volume from an image 2. boot an instance from the volume 3. create a snapshot from the instance 4. delete the volume snapshot 5. create a new snapshot from the instance scenario#2: 1. create a volume from an image 2. boot an instance from the volume 3. create a snapshot through the volume using --force=True 4. delete the volume 5. create a second snapshot for the volume using --force=True Actual results: we fail to create the snapshot with No such file or directory\n error Expected results: we should be able to create the snapshot Additional info: **we also cannot delete the snapshots with same error** 2013-11-22 16:11:36.955 14458 WARNING cinder.quota [req-66ebb212-b114-48c4-9f0c-19aafeb38918 24b77982be8049ee9cd5ad7bed913565 7eb59aa89e8944d098554ff6f5a4cf88] Deprecated: Default quota for resource: gigabytes is set by the default quota flag: quota_gigabytes, it is now deprecated. Please use the the default quota class for default quota. 2013-11-22 16:11:36.956 14458 WARNING cinder.quota [req-66ebb212-b114-48c4-9f0c-19aafeb38918 24b77982be8049ee9cd5ad7bed913565 7eb59aa89e8944d098554ff6f5a4cf88] Deprecated: Default quota for resource: snapshots is set by the default quota flag: quota_snapshots, it is now deprecated. Please use the the default quota class for default quota. 2013-11-22 16:11:52.879 14458 ERROR cinder.openstack.common.rpc.amqp [req-b7df5de9-3c2c-4a6f-9ca4-c2d442fafe48 24b77982be8049ee9cd5ad7bed913565 7eb59aa89e8944d098554ff6f5a4cf88] Exception during message handling 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp **args) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 371, in create_snapshot 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp {'status': 'error'}) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp self.gen.next() 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 362, in create_snapshot 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_snapshot(snapshot_ref) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 375, in create_snapshot 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp new_snap_path) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 464, in _create_qcow2_snap_file 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp self._execute(*command, run_as_root=True) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp return processutils.execute(*cmd, **kwargs) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 173, in execute 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp cmd=' '.join(cmd)) 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command. 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img create -f qcow2 -o backing_file=/var/lib/cinder/mnt/92ce777f40909398918e29a4128dfce0/volume-7e3df44e-14e1-4a92-b612-b0dd7731a4e2.013312d5-e64b-477b-a3ee-6a0b764a5f46 /var/lib/cinder/mnt/92ce777f40909398918e29a4128dfce0/volume-7e3df44e-14e1-4a92-b612-b0dd7731a4e2.aa750eab-31dc-4446-b0f0-2ed9ebd77ee2 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp Exit code: 1 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp Stdout: '' 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp Stderr: "Could not open '/var/lib/cinder/mnt/92ce777f40909398918e29a4128dfce0/volume-7e3df44e-14e1-4a92-b612-b0dd7731a4e2.013312d5-e64b-477b-a3ee-6a0b764a5f46': No such file or directory\n" 2013-11-22 16:11:52.879 14458 TRACE cinder.openstack.common.rpc.amqp (END)