Bug 1037772

Summary: ThinLVMVolumeDriver fails to create volume from snapshot
Product: Red Hat OpenStack Reporter: Eric Harney <eharney>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED CURRENTRELEASE QA Contact: Haim <hateya>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.0CC: eharney, gfidente, hateya, jobernar, yeylon
Target Milestone: asyncKeywords: ZStream
Target Release: 3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1031766 Environment:
Last Closed: 2013-12-18 16:17:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1031766    
Bug Blocks:    

Description Eric Harney 2013-12-03 19:11:17 UTC 
+++ This bug was initially created as a clone of Bug #1031766 +++

Description of problem:
when using the hinlvm driver, creating a new volume from snapshot fails as it looks for the snapshot outside the pool

2013-11-18 19:35:10    ERROR [cinder.volume.manager] volume volume-53a38c82-b67d-47f7-99bd-d348d567e91d: create failed
2013-11-18 19:35:10    ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 288, in create_volume
    LOG.error(_("volume %s: create failed"), volume_ref['name'])
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    self.gen.next()
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 268, in create_volume
    allow_reschedule)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 250, in create_volume
    image_location)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 172, in _create_volume
    snapshot_ref)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 160, in create_volume_from_snapshot
    snapshot['volume_size'])
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 117, in _copy_volume
    *extra_flags, run_as_root=True)
  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062 of=/dev/mapper/cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d count=1024 bs=1M
Exit code: 1
Stdout: ''
Stderr: "/bin/dd: opening `/dev/mapper/cinder--volumes-_snapshot--ce442c8e--473b--421d--8088--839901e49062': No such file or directory\n"


Version-Release number of selected component (if applicable):
openstack-cinder-2013.1.4-4.el6ost.noarch


Steps to Reproduce:
1. configure cinder for thinlvm
2. create a volume
3. create a volume snap
4. create a new volume from step created at #3

--- Additional comment from Eric Harney on 2013-11-18 12:47:15 EST ---

This sounds like the same issue that cropped up in bug 1023961 related to RHEL 6.5 LVM changes.

Can you paste "ls /dev/mapper/" and "lvs" output?

--- Additional comment from Eric Harney on 2013-11-18 12:48:54 EST ---

Cinder likely needs code to activate the LV before attempting the copy operation w/ dd.

--- Additional comment from Giulio Fidente on 2013-11-18 13:08:32 EST ---

hi Eric,

# ls /dev/mapper/
cinder--volumes-cinder--volumes--pool
cinder--volumes-cinder--volumes--pool-tpool
cinder--volumes-cinder--volumes--pool_tdata
cinder--volumes-cinder--volumes--pool_tmeta
cinder--volumes-volume--53a38c82--b67d--47f7--99bd--d348d567e91d
cinder--volumes-volume--aa3a58aa--c9cf--46fe--a10f--33c843ff47b8
control

# lvs
  LV                                             VG             Attr       LSize  Pool                Origin

  _snapshot-ce442c8e-473b-421d-8088-839901e49062 cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8

  cinder-volumes-pool                            cinder-volumes twi-a-tz-- 15.98g

  volume-53a38c82-b67d-47f7-99bd-d348d567e91d    cinder-volumes Vwi-a-tz--  1.00g cinder-volumes-pool
     
  volume-aa3a58aa-c9cf-46fe-a10f-33c843ff47b8    cinder-volumes Vwi-aotz--  1.00g cinder-volumes-pool

--- Additional comment from Giulio Fidente on 2013-12-03 10:10:18 EST ---

this also affects the --source-volid option which clones an existing volume:

2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 257, in create_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     flow.run(context.elevated())
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/decorators.py", line 105, in wrapper
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     return f(self, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 232, in run
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     run_it(r)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 212, in run_it
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.rollback(context, cause)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     result = runner(context, *args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/taskflow/utils.py", line 260, in __call__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.result = self.task(*args, **kwargs)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1499, in __call__
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     **volume_spec)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1339, in _create_from_source
_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 316, in create_cloned_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.delete_snapshot(temp_snapshot)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     self.clear_volume(volume, is_snapshot)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp     raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 17:05:09.778 29325 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/ma
pper/cinder--volumes-clone--snap--92c88aeb--25fb--4ff7--80a3--1903aa62e47f does not exist.

--- Additional comment from Giulio Fidente on 2013-12-03 11:24:32 EST ---

failing at time of a snapshot delete request too:

2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 424, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error_deleting'})
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 412, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.driver.delete_snapshot(snapshot_ref)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 265, in delete_snapshot
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self._delete_volume(snapshot, is_snapshot=True)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 136, in _delete_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     self.clear_volume(volume, is_snapshot)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 216, in clear_volume
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp     raise exception.VolumeBackendAPIException(data=msg)
2013-12-03 18:22:42.691 8985 TRACE cinder.openstack.common.rpc.amqp VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder--volumes-_snapshot--07257dea--ed80--4fdd--944d--043fe9d274d2 does not exist.

--- Additional comment from Eric Harney on 2013-12-03 14:09:08 EST ---

Moving this bug 4.0, will clone to a new 3.0.z bug.
Comment 1 Eric Harney 2013-12-18 16:17:07 UTC 
ThinLVM in RHOS3 is affected by this and other issues such as bug 1030080 and bug 1022714, among others.

Suggestion is to use RHOS4 for ThinLVM support.