Bug 1031766 - ThinLVMVolumeDriver fails to create volume from snapshot
Summary: ThinLVMVolumeDriver fails to create volume from snapshot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 3.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 4.0
Assignee: Eric Harney
QA Contact: Giulio Fidente
URL:
Whiteboard:
Depends On: 1037781
Blocks: 975606 998617 1037772
TreeView+ depends on / blocked
 
Reported: 2013-11-18 17:39 UTC by Giulio Fidente
Modified: 2016-04-27 01:44 UTC (History)
6 users (show)

Fixed In Version: openstack-cinder-2013.2-5.el6ost
Doc Type: Bug Fix
Doc Text:
Prior to Red Hat Enterprise Linux 6.5, the Logical Volume Manager (LVM) automatically activated Thin Provisioned Logical Volumes when needed. Because of this, the Block Storage service did not need to check if a logical volume was activated before reading from or writing to it. As of Red Hat Enterprise Linux 6.5, LVM no longer activates Thin Provisioned Logical Volumes automatically when needed. This prevented users from cloning volumes from snapshots on Block Storage services with Thin Provisioned Logical Volumes enabled. To address this, Block Storage services now activate logical volumes first (if required) before read/write operations.
Clone Of:
: 1037772 (view as bug list)
Environment:
Last Closed: 2013-12-20 00:36:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1252423 0 None None None Never
OpenStack gerrit 57015 0 None None None Never
Red Hat Bugzilla 1031766 0 unspecified CLOSED ThinLVMVolumeDriver fails to create volume from snapshot 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2013:1859 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory 2013-12-21 00:01:48 UTC

Internal Links: 1031766 1045428

Description Giulio Fidente 2013-11-18 17:39:12 UTC
Description of problem:
when using the hinlvm driver, creating a new volume from snapshot fails as it looks for the snapshot outside the pool

2013-11-18 19:35:10    ERROR [cinder.volume.manager] volume volume-53a38c82-b67d-47f7-99bd-d348d567e91d: create failed
2013-11-18 19:35:10    ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 288, in create_volume
    LOG.error(_("volume %s: create failed"), volume_ref['name'])
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    self.gen.next()
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 268, in create_volume
    allow_reschedule)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 250, in create_volume
    image_location)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 172, in _create_volume
    snapshot_ref)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 160, in create_volume_from_snapshot
    snapshot['volume_size'])
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 117, in _copy_volume
    *extra_flags, run_as_root=True)
  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062 of=/dev/mapper/cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d count=1024 bs=1M
Exit code: 1
Stdout: ''
Stderr: "/bin/dd: opening `/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062': No such file or directory\n"


Version-Release number of selected component (if applicable):
openstack-cinder-2013.1.4-4.el6ost.noarch


Steps to Reproduce:
1. configure cinder for thinlvm
2. create a volume
3. create a volume snap
4. create a new volume from step created at #3

Comment 1 Eric Harney 2013-11-18 17:47:15 UTC
This sounds like the same issue that cropped up in bug 1023961 related to RHEL 6.5 LVM changes.

Can you paste "ls /dev/mapper/" and "lvs" output?

Comment 2 Eric Harney 2013-11-18 17:48:54 UTC
Cinder likely needs code to activate the LV before attempting the copy operation w/ dd.

Comment 3 Giulio Fidente 2013-11-18 18:08:32 UTC
hi Eric,

# ls /dev/mapper/
cinder--volumes-cinder--volumes--pool
cinder--volumes-cinder--volumes--pool-tpool
cinder--volumes-cinder--volumes--pool_tdata
cinder--volumes-cinder--volumes--pool_tmeta
cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d
cinder--volumes-volume--aa3a58aa--c9cf--46fe--a10f--33c843ff47b8
control

# lvs
  LV                                             VG             Attr       LSize  Pool                Origin

  _snapshot-ce442c8e-473b-421d-8088-839901e49062 cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8

  cinder-volumes-pool                            cinder-volumes twi-a-tz-- 15.98g

  volume-53a38c82-b67d-47f7-99bd-d348d567e91d    cinder-volumes Vwi-a-tz--  1.00g cinder-volumes-pool
     
  volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8    cinder-volumes Vwi-aotz--  1.00g cinder-volumes-pool

Comment 4 Giulio Fidente 2013-12-03 15:10:18 UTC
this also affects the --source-volid option which clones an existing volume:

2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     flow.run(context.elevated())
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return f(self, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     run_it(r)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.rollback(context, cause)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     result = runner(context, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.result = self.task(*args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     **volume_spec)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source
_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 316, in create_cloned_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.delete_snapshot(temp_snapshot)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.clear_volume(volume, is_snapshot)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/ma
pper/cinder--volumes-clone--snap--92c88aeb--25fb--4ff7--80a3--1903aa62e47f does not exist.

Comment 5 Giulio Fidente 2013-12-03 16:24:32 UTC
failing at time of a snapshot delete request too:

2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 424, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error_deleting'})
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 412, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.driver.delete_snapshot(snapshot_ref)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.clear_volume(volume, is_snapshot)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder--volumes-_snapshot--07257dea--ed80--4fdd--944d--043fe9d274d2 does not exist.

Comment 6 Eric Harney 2013-12-03 19:09:08 UTC
Moving this bug 4.0, will clone to a new 3.0.z bug.

Comment 7 Eric Harney 2013-12-03 19:40:50 UTC
Giulio: those additional failures are happening because RHOS4 is missing a volume_clear patch that I thought was present.  Wrote bug 1037781.

Comment 9 Giulio Fidente 2013-12-11 17:08:00 UTC
verified using openstack-cinder-2013.2-8.el6ost.noarch

Comment 11 errata-xmlrpc 2013-12-20 00:36:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html


Note You need to log in before you can comment on or make changes to this bug.