Bug 1031766
| Summary: | ThinLVMVolumeDriver fails to create volume from snapshot | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Giulio Fidente <gfidente> | |
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> | |
| Status: | CLOSED ERRATA | QA Contact: | Giulio Fidente <gfidente> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 3.0 | CC: | ddomingo, eharney, gfidente, hateya, jobernar, yeylon | |
| Target Milestone: | rc | |||
| Target Release: | 4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-cinder-2013.2-5.el6ost | Doc Type: | Bug Fix | |
| Doc Text: |
Prior to Red Hat Enterprise Linux 6.5, the Logical Volume Manager (LVM) automatically activated Thin Provisioned Logical Volumes when needed. Because of this, the Block Storage service did not need to check if a logical volume was activated before reading from or writing to it.
As of Red Hat Enterprise Linux 6.5, LVM no longer activates Thin Provisioned Logical Volumes automatically when needed. This prevented users from cloning volumes from snapshots on Block Storage services with Thin Provisioned Logical Volumes enabled.
To address this, Block Storage services now activate logical volumes first (if required) before read/write operations.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1037772 (view as bug list) | Environment: | ||
| Last Closed: | 2013-12-20 00:36:59 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1037781 | |||
| Bug Blocks: | 975606, 998617, 1037772 | |||
This sounds like the same issue that cropped up in bug 1023961 related to RHEL 6.5 LVM changes. Can you paste "ls /dev/mapper/" and "lvs" output? Cinder likely needs code to activate the LV before attempting the copy operation w/ dd. hi Eric,
# ls /dev/mapper/
cinder--volumes-cinder--volumes--pool
cinder--volumes-cinder--volumes--pool-tpool
cinder--volumes-cinder--volumes--pool_tdata
cinder--volumes-cinder--volumes--pool_tmeta
cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d
cinder--volumes-volume--aa3a58aa--c9cf--46fe--a10f--33c843ff47b8
control
# lvs
LV VG Attr LSize Pool Origin
_snapshot-ce442c8e-473b-421d-8088-839901e49062 cinder-volumes Vwi---tz-k 1.00g cinder-volumes-pool volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8
cinder-volumes-pool cinder-volumes twi-a-tz-- 15.98g
volume-53a38c82-b67d-47f7-99bd-d348d567e91d cinder-volumes Vwi-a-tz-- 1.00g cinder-volumes-pool
volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8 cinder-volumes Vwi-aotz-- 1.00g cinder-volumes-pool
this also affects the --source-volid option which clones an existing volume: 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp **args) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp flow.run(context.elevated()) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp return f(self, *args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp run_it(r) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.rollback(context, cause) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.gen.next() 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp result = runner(context, *args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__ 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.result = self.task(*args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__ 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp **volume_spec) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source _volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 316, in create_cloned_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.delete_snapshot(temp_snapshot) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self._delete_volume(snapshot, is_snapshot=True) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.clear_volume(volume, is_snapshot) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp raise exception.VolumeBackendAPIException(data=msg) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/ma pper/cinder--volumes-clone--snap--92c88aeb--25fb--4ff7--80a3--1903aa62e47f does not exist. failing at time of a snapshot delete request too:
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp **args)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 424, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'})
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 412, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self.driver.delete_snapshot(snapshot_ref)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self.clear_volume(volume, is_snapshot)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder--volumes-_snapshot--07257dea--ed80--4fdd--944d--043fe9d274d2 does not exist.
Giulio: those additional failures are happening because RHOS4 is missing a volume_clear patch that I thought was present. Wrote bug 1037781. verified using openstack-cinder-2013.2-8.el6ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html |
Description of problem: when using the hinlvm driver, creating a new volume from snapshot fails as it looks for the snapshot outside the pool 2013-11-18 19:35:10 ERROR [cinder.volume.manager] volume volume-53a38c82-b67d-47f7-99bd-d348d567e91d: create failed 2013-11-18 19:35:10 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data rval = self.proxy.dispatch(ctxt, version, method, **args) File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch return getattr(proxyobj, method)(ctxt, **kwargs) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 288, in create_volume LOG.error(_("volume %s: create failed"), volume_ref['name']) File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ self.gen.next() File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 268, in create_volume allow_reschedule) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 250, in create_volume image_location) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 172, in _create_volume snapshot_ref) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 160, in create_volume_from_snapshot snapshot['volume_size']) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 117, in _copy_volume *extra_flags, run_as_root=True) File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute cmd=' '.join(cmd)) ProcessExecutionError: Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062 of=/dev/mapper/cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d count=1024 bs=1M Exit code: 1 Stdout: '' Stderr: "/bin/dd: opening `/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062': No such file or directory\n" Version-Release number of selected component (if applicable): openstack-cinder-2013.1.4-4.el6ost.noarch Steps to Reproduce: 1. configure cinder for thinlvm 2. create a volume 3. create a volume snap 4. create a new volume from step created at #3