Description of problem:
It is not possible to remove a volume e.g. test-pvc if that volume was used to
* create a snapshot e.g. test-pvc-snap (that was removed later)
* create a volume from that snapshot e.g. test-pvc-from-snap
Until you remove the volume from that snapshot (test-pvc-from-snap).
openstack client returns 0 but cinder-volume logs returns:
2021-08-03 13:09:28.616 135 DEBUG cinder.volume.drivers.rbd [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:1038
2021-08-03 13:09:28.621 135 ERROR cinder.volume.manager [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] Unable to delete busy volume.: cinder.exception.VolumeIsBusy: deleting volume volume-22a31673-e685-46ef-9656-382c8dca8357 that has snapshot
Version-Release number of selected component (if applicable):
* OSP 16.2-beta
* tested with and without rbd_flatten_volume_from_snapshot enabled (no difference was observed)
* Ceph cluster configured with 'rbd default clone format = 2'
* ceph osd set-require-min-compat-client mimic
Ceph + Cinder containers restarted.
Steps to Reproduce:
Create a volume:
openstack volume create --size 1 test-pvc
Create a snapshot:
openstack volume snapshot create --volume test-pvc test-snapshot-pvc
Create a volume from the snapshot:
openstack volume create --snapshot test-snapshot-pvc test-pvc-clone
Try deleting the initial volume, it fails as expected:
openstack volume delete test-pvc
Delete the snapshot:
openstack volume snapshot delete test-snapshot-pvc
Now, retry to delete the initial volume: it doesn't report any error:
openstack volume delete test-pvc
But the volume is still here and available:
openstack volume list
cinder-volume logs:
2021-08-03 13:09:28.616 135 DEBUG cinder.volume.drivers.rbd [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:1038
2021-08-03 13:09:28.621 135 ERROR cinder.volume.manager [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] Unable to delete busy volume.: cinder.exception.VolumeIsBusy: deleting volume volume-22a31673-e685-46ef-9656-382c8dca8357 that has snapshot
Now, delete the volume clone (created from a snapshot):
openstack volume delete test-pvc-clone
And retry to delete the initial volume:
openstack volume delete test-pvc
It worked, the volume was actually removed:
openstack volume list
Actual results:
openstackclient returns 0 but the volume isn't removed, an ERROR is seen in the logs.
We can see in the logs that Clone v2 is used:
"Using v2 Clone API"
Expected results:
The volume should be removed.
Additional info:
This was supposed to be fixed by https://bugzilla.redhat.com/show_bug.cgi?id=1764324
but we found the issue while investigating https://bugzilla.redhat.com/show_bug.cgi?id=1917710
The V2 CLONE API makes it impossible to delete the snapshot without removing the original volume test-pvc. However, in order to delete the cloned volume, cinder expects an extra configuration: rbd_flatten_volume_from_snapshot [1]. This config is set to False by default. To solve this issue enable it on cinder.conf.
[1] https://opendev.org/openstack/cinder/src/branch/master/cinder/volume/drivers/rbd.py#L1073