Bug 1023961 - ThinLVMVolumeDriver fails when deleting snapshots (RHEL 6.5)
ThinLVMVolumeDriver fails when deleting snapshots (RHEL 6.5)
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
Unspecified Unspecified
urgent Severity urgent
: z3
: 3.0
Assigned To: Eric Harney
: Regression, ZStream
Depends On:
Blocks: 1037781
  Show dependency treegraph
Reported: 2013-10-28 08:57 EDT by Giulio Fidente
Modified: 2016-04-26 18:46 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-cinder-2013.1.4-4.el6ost
Doc Type: Bug Fix
Doc Text:
Cause: Bug in ThinLVM driver caused by an incomplete backport to upstream stable/grizzly branch caused regression in upstream 2013.1.4 Consequence: ThinLVM volume driver attempts to clear volume data using an incorrect path which does not exist, so snapshot deletion will fail if volume clearing is enabled. (Enabled by default.) Fix: Make a stable/grizzly-only fix for the ThinLVM volume driver Result: Snapshot deletion now works with the ThinLVM volume driver -second issue- Cause: LVM behavior changed in RHEL 6.5 so that Thin snapshot volumes do not automatically have devices generated in /dev/mapper/. Consequence: Snapshot deletion fails for ThinLVM because it tries to wipe a device that does not exist. Fix: Disable wiping for ThinLVM as has been done for Havana. Result: Cinder snapshot operations now behave as expected.
Story Points: ---
Clone Of:
: 1037781 (view as bug list)
Last Closed: 2013-11-18 10:19:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1240299 None None None Never
Launchpad 1245529 None None None Never
OpenStack gerrit 54435 None None None Never
OpenStack gerrit 56083 None None None Never
Red Hat Product Errata RHBA-2013:1510 normal SHIPPED_LIVE Red Hat OpenStack 3.0 bug fix and enhancement advisory 2013-11-18 15:11:18 EST

  None (edit)
Description Giulio Fidente 2013-10-28 08:57:55 EDT
Description of problem:
when using the ThinLVMVolumeDriver driver, Cinder fails at deleting snapshots; the traceback in volume.log is as follows:
2013-10-28 14:54:31    ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 524, in delete_snapshot
    {'status': 'error_deleting'})
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 513, in delete_snapshot
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 249, in delete_snapshot
    self._delete_volume(snapshot, is_snapshot=True)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 132, in _delete_volume
    self.clear_volume(volume, is_snapshot=is_snapshot)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/lvm.py", line 200, in clear_volume
    raise exception.VolumeBackendAPIException(data=msg)
VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /dev/mapper/cinder--volumes-_snapshot--6956b504--375e--4eec--bcf7--aae91490c
219-cow does not exist.

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1. configure cinder with thinlvm driver
2. create a volume
3. create a volume snapshot
4. delete the snapshot

Actual results:
the snapshot is not deleted and found in "error_deleting" state
Comment 1 Eric Harney 2013-10-28 10:58:51 EDT
Appears to be unintended consequence of upstream backport of "0a86ef8 Fix secure delete for thick LVM snapshots" into stable/grizzly.  Logic changed from "if device does not exist, skip wiping" to "if device does not exist, fail" which is correct for thick LVM configuration, but breaks ThinLVM as the -cow device will never exist.
Comment 2 Ayal Baron 2013-10-29 06:21:58 EDT
Eric, does this happen in 4.0/Havana as well?
Comment 3 Eric Harney 2013-10-29 09:01:58 EDT
(In reply to Ayal Baron from comment #2)
> Eric, does this happen in 4.0/Havana as well?

No, Havana and newer skips running the volume wipe based on the lvm_type=thin option.  But this option is not present in Grizzly, so we need a fix to not run this code when volume_driver=...ThinLVMVolumeDriver.
Comment 4 Jon Bernard 2013-11-04 08:22:17 EST
I can help out on this if needed.  I'm not yet familiar with the backport process but I'm happy to give it a go.
Comment 10 Yogev Rabl 2013-11-11 09:29:11 EST
the bug wasn't fixed: 

version: openstack-cinder-2013.1.4-3.el6ost.noarch

the error is the same error as before.
Comment 11 Eric Harney 2013-11-11 10:44:15 EST
Yogev: Can you provide any more information?  I just tried this w/ the 2013.1.4-3 pkgs and it seems to work for me.  (Grizzly wipes w/ shred instead of dd for this snapshot wipe process.)

2013-11-11 10:41:08     INFO [cinder.volume.manager] snapshot snapshot-444c0fd8-fd02-4ae4-9ba9-a50421fd5053: deleting
2013-11-11 10:41:08    DEBUG [cinder.volume.manager] snapshot snapshot-444c0fd8-fd02-4ae4-9ba9-a50421fd5053: deleting
2013-11-11 10:41:08    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvdisplay cinder-volumes/_snapshot-444c0fd8-fd02-4ae4-9ba9-a50421fd5053
2013-11-11 10:41:09     INFO [cinder.volume.drivers.lvm] Performing secure delete on volume: 444c0fd8-fd02-4ae4-9ba9-a50421fd5053
2013-11-11 10:41:09    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf shred -n0 -z -s50MiB /dev/mapper/cinder--volumes-_snapshot--444c0fd8--fd02--4ae4--9ba9--a50421fd5053
2013-11-11 10:43:29    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvremove -f cinder-volumes/_snapshot-444c0fd8-fd02-4ae4-9ba9-a50421fd5053
Comment 13 Yogev Rabl 2013-11-11 17:12:38 EST
I installation configuration is: 

2 machines: 
1. VM with Cinder
2. A physical machine with all of the other components. 

the OS is RHEL 6.5. 

The cinder-volumes VG has a PV behind it (not the default loop0). 

if there's anything else, please ask.
Comment 15 Eric Harney 2013-11-12 13:25:18 EST
For some reason, LVM on Yogev's Cinder node is not populating /dev/mapper/ with entries for Cinder snapshots, only volumes.  Still looking into why.

My Cinder machine:

# cinder snapshot-list
| 36502cdd-bcea-400d-bfab-e7fbd2400d87 | a41927d7-d801-4a02-b659-f6e87926c578 | available |     None     |  1   |

# lvs | grep snapshot
  _snapshot-36502cdd-bcea-400d-bfab-e7fbd2400d87 cinder-volumes Vwi-a-tz-   1.00g cinder-volumes-pool volume-a41927d7-d801-4a02-b659-f6e87926c578   0.00 

# ls /dev/mapper/*snapshot*

Your Cinder machine:
# lvs | grep snapshot
  _snapshot-0021eb55-3a62-4f9c-b393-5f6326db3439 cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-5005b4b7-06dc-4f07-883e-af0b24d9f975                                 
  _snapshot-30a621aa-0799-435a-b43e-993f3a5eebe8 cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-469ba638-8d7a-4dde-be22-559f1ff5bc5c                                 
  _snapshot-31443b11-4f9a-472f-b59f-03f7cf1d33de cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-78ec04ac-1860-4aca-9c96-0cf0ddaa7b98                                 
  _snapshot-6d079948-a9e2-41df-a900-6a313d9ba276 cinder-volumes Vwi---tz-k  1.00g cinder-volumes-pool volume-469ba638-8d7a-4dde-be22-559f1ff5bc5c   
# ls /dev/mapper/*cinder*
/dev/mapper/cinder--volumes-cinder--volumes--pool        /dev/mapper/cinder--volumes-volume--469ba638--8d7a--4dde--be22--559f1ff5bc5c
/dev/mapper/cinder--volumes-cinder--volumes--pool_tdata  /dev/mapper/cinder--volumes-volume--5005b4b7--06dc--4f07--883e--af0b24d9f975
/dev/mapper/cinder--volumes-cinder--volumes--pool_tmeta  /dev/mapper/cinder--volumes-volume--78ec04ac--1860--4aca--9c96--0cf0ddaa7b98

# ls /dev/mapper/*snapshot*
ls: cannot access /dev/mapper/*snapshot*: No such file or directory
Comment 17 Eric Harney 2013-11-12 13:55:10 EST
Notable item from RHEL 6.5 lvm2 changelog:
  Automatically flag thin snapshots to be skipped during activation.
Comment 20 Yogev Rabl 2013-11-14 06:46:08 EST
verified on version: openstack-cinder-2013.1.4-4.el6ost.noarch
Comment 21 errata-xmlrpc 2013-11-18 10:19:32 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.