Bug 1031766 - ThinLVMVolumeDriver fails to create volume from snapshot
ThinLVMVolumeDriver fails to create volume from snapshot
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
3.0
Unspecified Unspecified
unspecified Severity medium
: rc
: 4.0
Assigned To: Eric Harney
Giulio Fidente
:
Depends On: 1037781
Blocks: 975606 998617 1037772
  Show dependency treegraph
 
Reported: 2013-11-18 12:39 EST by Giulio Fidente
Modified: 2016-04-26 21:44 EDT (History)
6 users (show)

See Also:
Fixed In Version: openstack-cinder-2013.2-5.el6ost
Doc Type: Bug Fix
Doc Text:
Prior to Red Hat Enterprise Linux 6.5, the Logical Volume Manager (LVM) automatically activated Thin Provisioned Logical Volumes when needed. Because of this, the Block Storage service did not need to check if a logical volume was activated before reading from or writing to it. As of Red Hat Enterprise Linux 6.5, LVM no longer activates Thin Provisioned Logical Volumes automatically when needed. This prevented users from cloning volumes from snapshots on Block Storage services with Thin Provisioned Logical Volumes enabled. To address this, Block Storage services now activate logical volumes first (if required) before read/write operations.
Story Points: ---
Clone Of:
: 1037772 (view as bug list)
Environment:
Last Closed: 2013-12-19 19:36:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1031766 None None None Never
Launchpad 1252423 None None None Never
OpenStack gerrit 57015 None None None Never

  None (edit)
Description Giulio Fidente 2013-11-18 12:39:12 EST
Description of problem:
when using the hinlvm driver, creating a new volume from snapshot fails as it looks for the snapshot outside the pool

2013-11-18 19:35:10    ERROR [cinder.volume.manager] volume volume-53a38c82-b67d-47f7-99bd-d348d567e91d: create failed
2013-11-18 19:35:10    ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 288, in create_volume
    LOG.error(_("volume %s: create failed"), volume_ref['name'])
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    self.gen.next()
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 268, in create_volume
    allow_reschedule)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 250, in create_volume
    image_location)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 172, in _create_volume
    snapshot_ref)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 160, in create_volume_from_snapshot
    snapshot['volume_size'])
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 117, in _copy_volume
    *extra_flags, run_as_root=True)
  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062 of=/dev/mapper/cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d count=1024 bs=1M
Exit code: 1
Stdout: ''
Stderr: "/bin/dd: opening `/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062': No such file or directory\n"


Version-Release number of selected component (if applicable):
openstack-cinder-2013.1.4-4.el6ost.noarch


Steps to Reproduce:
1. configure cinder for thinlvm
2. create a volume
3. create a volume snap
4. create a new volume from step created at #3
Comment 1 Eric Harney 2013-11-18 12:47:15 EST
This sounds like the same issue that cropped up in bug 1023961 related to RHEL 6.5 LVM changes.

Can you paste "ls /dev/mapper/" and "lvs" output?
Comment 2 Eric Harney 2013-11-18 12:48:54 EST
Cinder likely needs code to activate the LV before attempting the copy operation w/ dd.
Comment 3 Giulio Fidente 2013-11-18 13:08:32 EST
hi Eric,

# ls /dev/mapper/
cinder--volumes-cinder--volumes--pool
cinder--volumes-cinder--volumes--pool-tpool
cinder--volumes-cinder--volumes--pool_tdata
cinder--volumes-cinder--volumes--pool_tmeta
cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d
cinder--volumes-volume--aa3a58aa--c9cf--46fe--a10f--33c843ff47b8
control

# lvs
  LV                                             VG             Attr       LSize  Pool                Origin

  _snapshot-ce442c8e-473b-421d-8088-839901e49062 cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8

  cinder-volumes-pool                            cinder-volumes twi-a-tz-- 15.98g

  volume-53a38c82-b67d-47f7-99bd-d348d567e91d    cinder-volumes Vwi-a-tz--  1.00g cinder-volumes-pool
     
  volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8    cinder-volumes Vwi-aotz--  1.00g cinder-volumes-pool
Comment 4 Giulio Fidente 2013-12-03 10:10:18 EST
this also affects the --source-volid option which clones an existing volume:

2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     flow.run(context.elevated())
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return f(self, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     run_it(r)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.rollback(context, cause)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     result = runner(context, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.result = self.task(*args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     **volume_spec)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source
_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 316, in create_cloned_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.delete_snapshot(temp_snapshot)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.clear_volume(volume, is_snapshot)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/ma
pper/cinder--volumes-clone--snap--92c88aeb--25fb--4ff7--80a3--1903aa62e47f does not exist.
Comment 5 Giulio Fidente 2013-12-03 11:24:32 EST
failing at time of a snapshot delete request too:

2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 424, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error_deleting'})
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 412, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.driver.delete_snapshot(snapshot_ref)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.clear_volume(volume, is_snapshot)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder--volumes-_snapshot--07257dea--ed80--4fdd--944d--043fe9d274d2 does not exist.
Comment 6 Eric Harney 2013-12-03 14:09:08 EST
Moving this bug 4.0, will clone to a new 3.0.z bug.
Comment 7 Eric Harney 2013-12-03 14:40:50 EST
Giulio: those additional failures are happening because RHOS4 is missing a volume_clear patch that I thought was present.  Wrote bug 1037781.
Comment 9 Giulio Fidente 2013-12-11 12:08:00 EST
verified using openstack-cinder-2013.2-8.el6ost.noarch
Comment 11 errata-xmlrpc 2013-12-19 19:36:59 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.