Description of problem: when using the hinlvm driver, creating a new volume from snapshot fails as it looks for the snapshot outside the pool 2013-11-18 19:35:10 ERROR [cinder.volume.manager] volume volume-53a38c82-b67d-47f7-99bd-d348d567e91d: create failed 2013-11-18 19:35:10 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data rval = self.proxy.dispatch(ctxt, version, method, **args) File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch return getattr(proxyobj, method)(ctxt, **kwargs) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 288, in create_volume LOG.error(_("volume %s: create failed"), volume_ref['name']) File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ self.gen.next() File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 268, in create_volume allow_reschedule) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 250, in create_volume image_location) File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 172, in _create_volume snapshot_ref) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 160, in create_volume_from_snapshot snapshot['volume_size']) File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 117, in _copy_volume *extra_flags, run_as_root=True) File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute cmd=' '.join(cmd)) ProcessExecutionError: Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062 of=/dev/mapper/cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d count=1024 bs=1M Exit code: 1 Stdout: '' Stderr: "/bin/dd: opening `/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062': No such file or directory\n" Version-Release number of selected component (if applicable): openstack-cinder-2013.1.4-4.el6ost.noarch Steps to Reproduce: 1. configure cinder for thinlvm 2. create a volume 3. create a volume snap 4. create a new volume from step created at #3
This sounds like the same issue that cropped up in bug 1023961 related to RHEL 6.5 LVM changes. Can you paste "ls /dev/mapper/" and "lvs" output?
Cinder likely needs code to activate the LV before attempting the copy operation w/ dd.
hi Eric, # ls /dev/mapper/ cinder--volumes-cinder--volumes--pool cinder--volumes-cinder--volumes--pool-tpool cinder--volumes-cinder--volumes--pool_tdata cinder--volumes-cinder--volumes--pool_tmeta cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d cinder--volumes-volume--aa3a58aa--c9cf--46fe--a10f--33c843ff47b8 control # lvs LV VG Attr LSize Pool Origin _snapshot-ce442c8e-473b-421d-8088-839901e49062 cinder-volumes Vwi---tz-k 1.00g cinder-volumes-pool volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8 cinder-volumes-pool cinder-volumes twi-a-tz-- 15.98g volume-53a38c82-b67d-47f7-99bd-d348d567e91d cinder-volumes Vwi-a-tz-- 1.00g cinder-volumes-pool volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8 cinder-volumes Vwi-aotz-- 1.00g cinder-volumes-pool
this also affects the --source-volid option which clones an existing volume: 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp **args) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp flow.run(context.elevated()) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp return f(self, *args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp run_it(r) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.rollback(context, cause) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.gen.next() 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp result = runner(context, *args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__ 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.result = self.task(*args, **kwargs) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__ 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp **volume_spec) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source _volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 316, in create_cloned_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.delete_snapshot(temp_snapshot) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self._delete_volume(snapshot, is_snapshot=True) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp self.clear_volume(volume, is_snapshot) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp raise exception.VolumeBackendAPIException(data=msg) 2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/ma pper/cinder--volumes-clone--snap--92c88aeb--25fb--4ff7--80a3--1903aa62e47f does not exist.
failing at time of a snapshot delete request too: 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp **args) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 424, in delete_snapshot 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'}) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self.gen.next() 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 412, in delete_snapshot 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self.driver.delete_snapshot(snapshot_ref) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self._delete_volume(snapshot, is_snapshot=True) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp self.clear_volume(volume, is_snapshot) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp raise exception.VolumeBackendAPIException(data=msg) 2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder--volumes-_snapshot--07257dea--ed80--4fdd--944d--043fe9d274d2 does not exist.
Moving this bug 4.0, will clone to a new 3.0.z bug.
Giulio: those additional failures are happening because RHOS4 is missing a volume_clear patch that I thought was present. Wrote bug 1037781.
verified using openstack-cinder-2013.2-8.el6ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html