This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 1989680 - RBD Volume can't be removed if other volumes were created from a snapshot of this volume and were not removed [NEEDINFO]
Summary: RBD Volume can't be removed if other volumes were created from a snapshot of ...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 16.2 (Train)
Hardware: All
OS: All
medium
high
Target Milestone: ---
: ---
Assignee: Eric Harney
QA Contact: Evelina Shames
RHOS Documentation Team
URL:
Whiteboard:
: 2050163 (view as bug list)
Depends On: 2231960 1997715
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-03 17:40 UTC by Emilien Macchi
Modified: 2024-04-25 06:30 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2231960 (view as bug list)
Environment:
Last Closed: 2024-01-05 11:06:21 UTC
Target Upstream Version:
Embargoed:
enothen: needinfo? (udesale)

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1969643 0 None None None 2022-10-27 18:52:06 UTC
OpenStack gerrit 835384 0 None NEW RBD: Flattening of child volumes during deletion 2023-04-12 12:21:46 UTC
Red Hat Issue Tracker   OSP-6735 0 None None None 2024-04-25 06:29:59 UTC

Internal Links: 2117852

Description Emilien Macchi 2021-08-03 17:40:01 UTC 
Description of problem:
It is not possible to remove a volume e.g. test-pvc if that volume was used to
* create a snapshot e.g. test-pvc-snap (that was removed later)
* create a volume from that snapshot e.g. test-pvc-from-snap

Until you remove the volume from that snapshot (test-pvc-from-snap).
openstack client returns 0 but cinder-volume logs returns:

2021-08-03 13:09:28.616 135 DEBUG cinder.volume.drivers.rbd [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:1038
2021-08-03 13:09:28.621 135 ERROR cinder.volume.manager [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] Unable to delete busy volume.: cinder.exception.VolumeIsBusy: deleting volume volume-22a31673-e685-46ef-9656-382c8dca8357 that has snapshot

Version-Release number of selected component (if applicable):

* OSP 16.2-beta
* tested with and without rbd_flatten_volume_from_snapshot enabled (no difference was observed)
* Ceph cluster configured with 'rbd default clone format = 2'
* ceph osd set-require-min-compat-client mimic

Ceph + Cinder containers restarted.

Steps to Reproduce:

Create a volume:
openstack volume create --size 1 test-pvc
 
Create a snapshot:
openstack volume snapshot create --volume test-pvc test-snapshot-pvc
 
Create a volume from the snapshot:
openstack volume create --snapshot test-snapshot-pvc test-pvc-clone
 
Try deleting the initial volume, it fails as expected:
openstack volume delete test-pvc
 
Delete the snapshot:
openstack volume snapshot delete test-snapshot-pvc
 
Now, retry to delete the initial volume: it doesn't report any error:
openstack volume delete test-pvc
 
But the volume is still here and available:
openstack volume list
 
cinder-volume logs:
 
2021-08-03 13:09:28.616 135 DEBUG cinder.volume.drivers.rbd [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:1038
2021-08-03 13:09:28.621 135 ERROR cinder.volume.manager [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] Unable to delete busy volume.: cinder.exception.VolumeIsBusy: deleting volume volume-22a31673-e685-46ef-9656-382c8dca8357 that has snapshot
 
Now, delete the volume clone (created from a snapshot):
openstack volume delete test-pvc-clone
 
And retry to delete the initial volume:
openstack volume delete test-pvc
 
It worked, the volume was actually removed:
openstack volume list


Actual results:

openstackclient returns 0 but the volume isn't removed, an ERROR is seen in the logs.
We can see in the logs that Clone v2 is used:
"Using v2 Clone API"

Expected results:
The volume should be removed.

Additional info:
This was supposed to be fixed by https://bugzilla.redhat.com/show_bug.cgi?id=1764324
but we found the issue while investigating https://bugzilla.redhat.com/show_bug.cgi?id=1917710
Comment 2 Sofia Enriquez 2021-08-11 02:01:02 UTC 
The V2 CLONE API makes it impossible to delete the snapshot without removing the original volume test-pvc. However, in order to delete the cloned volume, cinder expects an extra configuration: rbd_flatten_volume_from_snapshot [1]. This config is set to False by default. To solve this issue enable it on cinder.conf.

[1] https://opendev.org/openstack/cinder/src/branch/master/cinder/volume/drivers/rbd.py#L1073
Comment 10 Eduard Barrera 2022-02-03 13:31:52 UTC 
*** Bug 2050163 has been marked as a duplicate of this bug. ***
Comment 15 rlobillo 2023-04-10 09:13:35 UTC 
For Openshift on Openstack, this bug is provoking the K8s being hung on PVC deletion. Rising the severity.
Comment 16 Sofia Enriquez 2023-04-26 16:06:51 UTC 
Hello

Currently there is a Cinder upstream patch in progress. We need changes in Glance similar to Cinder to allow deletion of parent images that have dependent volumes. Eric lead the discussion at Bobcat PTG https://wiki.openstack.org/wiki/CinderBobcatPTGSummary#RBD_deletion_issues

Cheers,
Note You need to log in before you can comment on or make changes to this bug.