Description of problem: When a child volume is created from the snapshot of the parent cinder volume, deleting the parent cinder volume produces the following misleading information: Issue 1.) HTTP method code 202 is returned (success) but parent volume is not deleted. Issue 2.) Ceilometer updates the gnocchi resource as ended as soon as delete action is triggered without validating if the resource is deleted. (This is causing billing issues, as the asset remains active. Version-Release number of selected component (if applicable): OSP 13 How reproducible: Customer can reproduce reliably. Steps to Reproduce: [1] $ openstack volume show 71fbc360-3275-49a3-b6e3-e3ed62369020 +--------------------------------+---------------------------------------------------------------+ | Field | Value | +--------------------------------+---------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-04-16T09:22:07.000000 | | description | None | | encrypted | False | | id | 71fbc360-3275-49a3-b6e3-e3ed62369020 | | migration_status | None | | multiattach | False | | name | bill_test_vol1 | | os-vol-host-attr:host | hostgroup@tripleo_ceph_volumes-sata#tripleo_ceph_volumes-sata | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 8885e368558d4ac6a63526b1e7ca0f5e | | properties | uai='UAI2008331' | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | type | ceph-sata | | updated_at | 2020-04-24T03:52:50.000000 | | user_id | 5b259fbb03384ddaa9b340ec7cf04f3c | $ openstack volume snapshot list --volume 71fbc360-3275-49a3-b6e3-e3ed62369020 --all +--------------------------------------+-----------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+-----------------+-------------+-----------+------+ | 23f50a7b-8672-439d-bc83-2bf2337a34cd | bill_test_snap1 | None | available | 1 | +--------------------------------------+-----------------+-------------+-----------+------+ ## Clone volume from snapshot $ osiad2-adm volume show e9d363af-c6ce-4242-a757-d52a8fce17ee IAD2:ADMIN +--------------------------------+---------------------------------------------------------------+ | Field | Value | +--------------------------------+---------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-04-16T09:23:28.000000 | | description | None | | encrypted | False | | id | e9d363af-c6ce-4242-a757-d52a8fce17ee | | migration_status | None | | multiattach | False | | name | bill_test_clone | | os-vol-host-attr:host | hostgroup@tripleo_ceph_volumes-sata#tripleo_ceph_volumes-sata | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 8885e368558d4ac6a63526b1e7ca0f5e | | properties | uai='UAI2008331' | | replication_status | None | | size | 1 | | snapshot_id | 23f50a7b-8672-439d-bc83-2bf2337a34cd | | source_volid | None | | status | available | | type | ceph-sata | | updated_at | 2020-04-16T09:23:29.000000 | | user_id | 5b259fbb03384ddaa9b340ec7cf04f3c | +--------------------------------+---------------------------------------------------------------+ ## At Ceph Layer Parent Volume : # rbd -p volumes-sata info volume-71fbc360-3275-49a3-b6e3-e3ed62369020 rbd image 'volume-71fbc360-3275-49a3-b6e3-e3ed62369020': size 1GiB in 256 objects order 22 (4MiB objects) block_name_prefix: rbd_data.85ae6848d6472d format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: create_timestamp: Thu Apr 16 09:22:07 2020 ## Snapshot of Parent Volume ]# rbd -p volumes-sata snap ls volume-71fbc360-3275-49a3-b6e3-e3ed62369020 SNAPID NAME SIZE TIMESTAMP 98108 snapshot-23f50a7b-8672-439d-bc83-2bf2337a34cd 1GiB Thu Apr 16 09:22:58 2020 ## Clone Volume from Snapshot [root@iad1-ceph-mon01 ~]# rbd -p volumes-sata children volume-71fbc360-3275-49a3-b6e3-e3ed62369020@snapshot-23f50a7b-8672-439d-bc83-2bf2337a34cd volumes-sata/volume-e9d363af-c6ce-4242-a757-d52a8fce17ee [root@iad1-ceph-mon01 ~]# [root@iad1-ceph-mon01 ~]# [root@iad1-ceph-mon01 ~]# rbd -p volumes-sata info volumes-sata/volume-e9d363af-c6ce-4242-a757-d52a8fce17ee rbd image 'volume-e9d363af-c6ce-4242-a757-d52a8fce17ee': size 1GiB in 256 objects order 22 (4MiB objects) block_name_prefix: rbd_data.86d3497b0f2264 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: create_timestamp: Thu Apr 16 09:23:29 2020 parent: volumes-sata/volume-71fbc360-3275-49a3-b6e3-e3ed62369020@snapshot-23f50a7b-8672-439d-bc83-2bf2337a34cd overlap: 1GiB [2] ## HTTP code && Debug Logs $ date && openstack --debug volume delete --purge 71fbc360-3275-49a3-b6e3-e3ed62369020 && date Fri Apr 24 06:12:44 UTC 2020 START with options: --debug volume delete --purge 71fbc360-3275-49a3-b6e3-e3ed62369020 ... REQ: curl -g -i -X GET https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020 -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}527755b93fb325013c2e0855842c467abb3d6af877cfd989f0e417406fe68382" Starting new HTTPS connection (1): us-east1.example.cloud.example.com https://us-east1.example.cloud.example.com:13776 "GET /v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020 HTTP/1.1" 200 541 RESP: [200] Content-Encoding: gzip Content-Length: 541 Content-Type: application/json Date: Fri, 24 Apr 2020 06:12:46 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version,Accept-Encoding x-compute-request-id: req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4 x-openstack-request-id: req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4 RESP BODY: {"volume": {"migration_status": null, "attachments": [], "links": [{"href": "https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020", "rel": "self"}, {"href": "https://us-east1.example.cloud.example.com:13776/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "hostgroup@tripleo_ceph_volumes-sata#tripleo_ceph_volumes-sata", "encrypted": false, "updated_at": "2020-04-24T03:52:50.000000", "replication_status": null, "snapshot_id": null, "id": "71fbc360-3275-49a3-b6e3-e3ed62369020", "size": 1, "user_id": "5b259fbb03384ddaa9b340ec7cf04f3c", "os-vol-tenant-attr:tenant_id": "8885e368558d4ac6a63526b1e7ca0f5e", "os-vol-mig-status-attr:migstat": null, "metadata": {"uai": "UAI2008331"}, "status": "available", "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "bill_test_vol1", "bootable": "false", "created_at": "2020-04-16T09:22:07.000000", "volume_type": "ceph-sata"}} GET call to volumev3 for https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020 used request id req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4 REQ: curl -g -i -X DELETE https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}527755b93fb325013c2e0855842c467abb3d6af877cfd989f0e417406fe68382" -H "X-OpenStack-Request-ID: req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4" https://us-east1.example.cloud.example.com:13776 "DELETE /v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True HTTP/1.1" 202 0 RESP: [202] Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 24 Apr 2020 06:12:46 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version x-openstack-request-id: req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 RESP BODY: Omitted, Content-Type is set to text/html; charset=UTF-8. Only application/json responses have their bodies logged. DELETE call to volumev3 for https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True used request id req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 clean_up DeleteVolume: END return value: 0 Fri Apr 24 06:12:47 UTC 2020 ## Parent Volume, Snapshot and Clone volume status $ date && openstack volume show 71fbc360-3275-49a3-b6e3-e3ed62369020 && openstack volume snapshot list --volume 71fbc360-3275-49a3-b6e3-e3ed62369020 --all && openstack volume show e9d363af-c6ce-4242-a757-d52a8fce17ee Fri Apr 24 06:16:19 UTC 2020 +--------------------------------+---------------------------------------------------------------+ | Field | Value | +--------------------------------+---------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-04-16T09:22:07.000000 | | description | None | | encrypted | False | | id | 71fbc360-3275-49a3-b6e3-e3ed62369020 | | migration_status | None | | multiattach | False | | name | bill_test_vol1 | | os-vol-host-attr:host | hostgroup@tripleo_ceph_volumes-sata#tripleo_ceph_volumes-sata | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 8885e368558d4ac6a63526b1e7ca0f5e | | properties | uai='UAI2008331' | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | type | ceph-sata | | updated_at | 2020-04-24T06:12:47.000000 | | user_id | 5b259fbb03384ddaa9b340ec7cf04f3c | +--------------------------------+---------------------------------------------------------------+ +--------------------------------------+-----------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+-----------------+-------------+-----------+------+ | 23f50a7b-8672-439d-bc83-2bf2337a34cd | bill_test_snap1 | None | available | 1 | +--------------------------------------+-----------------+-------------+-----------+------+ +--------------------------------+---------------------------------------------------------------+ | Field | Value | +--------------------------------+---------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-04-16T09:23:28.000000 | | description | None | | encrypted | False | | id | e9d363af-c6ce-4242-a757-d52a8fce17ee | | migration_status | None | | multiattach | False | | name | bill_test_clone | | os-vol-host-attr:host | hostgroup@tripleo_ceph_volumes-sata#tripleo_ceph_volumes-sata | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 8885e368558d4ac6a63526b1e7ca0f5e | | properties | uai='UAI2008331' | | replication_status | None | | size | 1 | | snapshot_id | 23f50a7b-8672-439d-bc83-2bf2337a34cd | | source_volid | None | | status | available | | type | ceph-sata | | updated_at | 2020-04-16T09:23:29.000000 | | user_id | 5b259fbb03384ddaa9b340ec7cf04f3c | +--------------------------------+-------------------------------------------------------------- ## Log: /var/log/containers/cinder/cinder-api.log:2020-04-24 06:12:46.869 26 INFO cinder.api.openstack.wsgi [req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 7b5516ddd8a9477388a1f4e8e0764fa2 3d769c76682347f597643ad3509b5354 - default default] DELETE https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True /var/log/containers/cinder/cinder-api.log:2020-04-24 06:12:46.870 26 INFO cinder.api.v3.volumes [req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 7b5516ddd8a9477388a1f4e8e0764fa2 3d769c76682347f597643ad3509b5354 - default default] Delete volume with id: 71fbc360-3275-49a3-b6e3-e3ed62369020 /var/log/containers/cinder/cinder-volume.log:2020-04-24 06:12:47.319 180 ERROR cinder.volume.manager [req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 7b5516ddd8a9477388a1f4e8e0764fa2 3d769c76682347f597643ad3509b5354 - default default] Unable to delete busy volume.: VolumeIsBusy: deleting volume volume-71fbc360-3275-49a3-b6e3-e3ed62369020 that has snapshot [3] ## Gnocchi Ended $ openstack metric resource show 71fbc360-3275-49a3-b6e3-e3ed62369020 +-----------------------+-------------------------------------------------------------------+ | Field | Value | +-----------------------+-------------------------------------------------------------------+ | created_by_project_id | a676070880324629a6d96883e7343883 | | created_by_user_id | 1d3ab353e7f74d2fac0453362d8240f9 | | creator | 1d3ab353e7f74d2fac0453362d8240f9:a676070880324629a6d96883e7343883 | | ended_at | 2020-04-24T06:13:15.537267+00:00 | <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< | id | 71fbc360-3275-49a3-b6e3-e3ed62369020 | | metrics | volume.backup.size: 3904e9a5-13a8-433f-a4f7-9cba8edecb99 | | | volume.size: 8c7c78c8-9bea-4cc2-9a8d-4c8e3c256da1 | | | volume.snapshot.size: a1b6cacb-d35a-4783-ac58-79bff7d57815 | | | volume: e6c7b887-6fcf-4ff7-986d-30b151b9be1d | | original_resource_id | 71fbc360-3275-49a3-b6e3-e3ed62369020 | | project_id | 8885e368558d4ac6a63526b1e7ca0f5e | | revision_end | None | | revision_start | 2020-04-24T06:13:16.409231+00:00 | | started_at | 2020-04-16T09:22:08.698356+00:00 | | type | volume | | user_id | 5b259fbb03384ddaa9b340ec7cf04f3c | +-----------------------+-------------------------------------------------------------------+ Actual results: Volume is reported as deleted, but it is in fact not deleted. Expected results: Return code matches the action, and the delete should succeed. Additional info: Issue is causing billing discrepancies because the asset is still in existence even though incorrectly reported as deleted.
This is a known aspect of RBD volumes, and the behavior can be modified using cinder's rbd_flatten_volume_from_snapshot parameter. The value defaults to False, which leads to the problem you're seeing. We recently enhanced OSP-16 with a new CinderRbdFlattenVolumeFromSnapshot THT parameter (see bug #1798917). For OSP-13, bug #1744855 comment #3 describes an alternate way of setting rbd_flatten_volume_from_snapshot in cinder.conf. To clarify what's happening, the request to delete the parent volume is accepted at the API layer, but the actual deletion is blocked after the request reaches the cinder-volume service. While it would be nice if that could be caught at the API layer, it's not feasible when you consider each cinder backend driver has its own set of constraints. The delete request is accepted (request looks good!), but is rejected later by RBD backend driver. *** This bug has been marked as a duplicate of bug 1744855 ***
Hi Alan, People may not want to flatten rbd images when creating volumes from snapshots, since loosing the CoW capability. It is perfectly fine for volume deletion to fail if fails on the backend ( I assume delete, this is a synchronous call to the cinder-volume/RBD driver) The bigger problem is the issue with gnocchi, which makes volumes not get billed after the delete action like that. Gnocchi should handle this correctly, or at least, should include a pollster to fix such Gnocchi resources. It may be just a 'feature' in Cinder, but certainly a bug in Gnocchi.
Hi Attila, I agree with everything you say. I don't work with gnocchi, but it sounds like it's responding to the volume delete request getting accepted by the API layer, and not waiting to see if the deletion completed successfully. That should probably be investigated as a different bug than this one.