Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1842571

Summary: Ceilometer updates the gnocchi resource as ended as soon as delete action is triggered without validating if the resource is deleted.
Product: Red Hat OpenStack Reporter: KOSAL RAJ I <kiyyappa>
Component: openstack-ceilometerAssignee: Matthias Runge <mrunge>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: apevec, jschluet, lhh, mrunge, pkilambi, rdiwakar
Target Milestone: z13Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ceilometer-10.0.1-12.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-28 18:31:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description KOSAL RAJ I 2020-06-01 14:42:16 UTC
Description of problem:
Ceilometer updates the gnocchi resource as ended as soon as delete action is triggered without validating if the resource is deleted.

Version-Release number of selected component (if applicable):
RHOSP13

Steps to Reproduce:

$ date && openstack --debug volume delete --purge 71fbc360-3275-49a3-b6e3-e3ed62369020 && date
Fri Apr 24 06:12:44 UTC 2020
START with options: --debug volume delete --purge 71fbc360-3275-49a3-b6e3-e3ed62369020
...
REQ: curl -g -i -X GET https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020 -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}527755b93fb325013c2e0855842c467abb3d6af877cfd989f0e417406fe68382"
Starting new HTTPS connection (1): us-east1.example.cloud.example.com
https://us-east1.example.cloud.example.com:13776 "GET /v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020 HTTP/1.1" 200 541
RESP: [200] Content-Encoding: gzip Content-Length: 541 Content-Type: application/json Date: Fri, 24 Apr 2020 06:12:46 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version,Accept-Encoding x-compute-request-id: req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4 x-openstack-request-id: req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4
RESP BODY: {"volume": {"migration_status": null, "attachments": [], "links": [{"href": "https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020", "rel": "self"}, {"href": "https://us-east1.example.cloud.example.com:13776/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "hostgroup@tripleo_ceph_volumes-sata#tripleo_ceph_volumes-sata", "encrypted": false, "updated_at": "2020-04-24T03:52:50.000000", "replication_status": null, "snapshot_id": null, "id": "71fbc360-3275-49a3-b6e3-e3ed62369020", "size": 1, "user_id": "5b259fbb03384ddaa9b340ec7cf04f3c", "os-vol-tenant-attr:tenant_id": "8885e368558d4ac6a63526b1e7ca0f5e", "os-vol-mig-status-attr:migstat": null, "metadata": {"uai": "UAI2008331"}, "status": "available", "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "bill_test_vol1", "bootable": "false", "created_at": "2020-04-16T09:22:07.000000", "volume_type": "ceph-sata"}}
GET call to volumev3 for https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020 used request id req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4
REQ: curl -g -i -X DELETE https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}527755b93fb325013c2e0855842c467abb3d6af877cfd989f0e417406fe68382" -H "X-OpenStack-Request-ID: req-0c7fd30d-466d-4250-a7fd-f89b0daa1db4"
https://us-east1.example.cloud.example.com:13776 "DELETE /v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True HTTP/1.1" 202 0
RESP: [202] Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 24 Apr 2020 06:12:46 GMT OpenStack-API-Version: volume 3.0 Server: Apache Vary: OpenStack-API-Version x-openstack-request-id: req-886ee9ca-8574-4ed0-aa58-ee18f2396c61
RESP BODY: Omitted, Content-Type is set to text/html; charset=UTF-8. Only application/json responses have their bodies logged.
DELETE call to volumev3 for https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True used request id req-886ee9ca-8574-4ed0-aa58-ee18f2396c61
clean_up DeleteVolume:
END return value: 0
Fri Apr 24 06:12:47 UTC 2020

## Log:
/var/log/containers/cinder/cinder-api.log:2020-04-24 06:12:46.869 26 INFO cinder.api.openstack.wsgi [req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 7b5516ddd8a9477388a1f4e8e0764fa2 3d769c76682347f597643ad3509b5354 - default default] DELETE https://us-east1.example.cloud.example.com:13776/v3/3d769c76682347f597643ad3509b5354/volumes/71fbc360-3275-49a3-b6e3-e3ed62369020?cascade=True
/var/log/containers/cinder/cinder-api.log:2020-04-24 06:12:46.870 26 INFO cinder.api.v3.volumes [req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 7b5516ddd8a9477388a1f4e8e0764fa2 3d769c76682347f597643ad3509b5354 - default default] Delete volume with id: 71fbc360-3275-49a3-b6e3-e3ed62369020

/var/log/containers/cinder/cinder-volume.log:2020-04-24 06:12:47.319 180 ERROR cinder.volume.manager [req-886ee9ca-8574-4ed0-aa58-ee18f2396c61 7b5516ddd8a9477388a1f4e8e0764fa2 3d769c76682347f597643ad3509b5354 - default default] Unable to delete busy volume.: VolumeIsBusy: deleting volume volume-71fbc360-3275-49a3-b6e3-e3ed62369020 that has snapshot

[3]
## Gnocchi Ended 
$ openstack metric resource show 71fbc360-3275-49a3-b6e3-e3ed62369020
+-----------------------+-------------------------------------------------------------------+
| Field                 | Value                                                             |
+-----------------------+-------------------------------------------------------------------+
| created_by_project_id | a676070880324629a6d96883e7343883                                  |
| created_by_user_id    | 1d3ab353e7f74d2fac0453362d8240f9                                  |
| creator               | 1d3ab353e7f74d2fac0453362d8240f9:a676070880324629a6d96883e7343883 |
| ended_at              | 2020-04-24T06:13:15.537267+00:00                                  |       <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
| id                    | 71fbc360-3275-49a3-b6e3-e3ed62369020                              |
| metrics               | volume.backup.size: 3904e9a5-13a8-433f-a4f7-9cba8edecb99          |
|                       | volume.size: 8c7c78c8-9bea-4cc2-9a8d-4c8e3c256da1                 |
|                       | volume.snapshot.size: a1b6cacb-d35a-4783-ac58-79bff7d57815        |
|                       | volume: e6c7b887-6fcf-4ff7-986d-30b151b9be1d                      |
| original_resource_id  | 71fbc360-3275-49a3-b6e3-e3ed62369020                              |
| project_id            | 8885e368558d4ac6a63526b1e7ca0f5e                                  |
| revision_end          | None                                                              |
| revision_start        | 2020-04-24T06:13:16.409231+00:00                                  |
| started_at            | 2020-04-16T09:22:08.698356+00:00                                  |
| type                  | volume                                                            |
| user_id               | 5b259fbb03384ddaa9b340ec7cf04f3c                                  |
+-----------------------+-------------------------------------------------------------------+

This is reference to the BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1829960

Comment 12 Matthias Runge 2020-06-10 18:46:27 UTC
I created a volume (one), a snapshot from that volume (one-snap), a volume (two) from the snapshot and tried to delete the volume two. In this case, I can not even delete with --purge, but at the same time, gnocchi does NOT mark the volume as ended. 
While not being able to delete a new volume created from a snapshot may be wrong, I don't see inconsistent behavior here in ceilometer.

Again, am I missing something?

Comment 24 Matthias Runge 2020-06-18 06:17:03 UTC
I've posted a fix on https://review.opendev.org/736445

Comment 35 Leonid Natapov 2020-10-20 05:30:07 UTC
(overcloud) [stack@undercloud-0 ~]$ openstack volume create --size 20 lala
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2020-10-20T05:27:00.000000           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | b6e4463b-463b-441d-9634-68f3d95b2ec8 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | lala                                 |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 20                                   |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | tripleo                              |
| updated_at          | None                                 |
| user_id             | 2fc8ac9126654aaa865bd2951dce769b     |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot create --volume b6e4463b-463b-441d-9634-68f3d95b2ec  foo_snap
No volume with a name or ID of 'b6e4463b-463b-441d-9634-68f3d95b2ec' exists.
(overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot create --volume b6e4463b-463b-441d-9634-68f3d95b2ec8  foo_snap
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| created_at  | 2020-10-20T05:27:42.571786           |
| description | None                                 |
| id          | fe76b453-b184-4dde-b2c0-1b553953de64 |
| name        | foo_snap                             |
| properties  |                                      |
| size        | 20                                   |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | b6e4463b-463b-441d-9634-68f3d95b2ec8 |
+-------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack volume create --snapshot fe76b453-b184-4dde-b2c0-1b553953de64 clone
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2020-10-20T05:28:05.000000           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 3b97ad9f-bfbc-46ae-8676-0949a6db4493 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | clone                                |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 20                                   |
| snapshot_id         | fe76b453-b184-4dde-b2c0-1b553953de64 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | tripleo                              |
| updated_at          | None                                 |
| user_id             | 2fc8ac9126654aaa865bd2951dce769b     |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack volume delete --purge b6e4463b-463b-441d-9634-68f3d95b2ec8
(overcloud) [stack@undercloud-0 ~]$ openstack metric resource show b6e4463b-463b-441d-9634-68f3d95b2ec8
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| created_by_project_id | e4342cc6968c430ea7467623eb6b97ee                                            |
| created_by_user_id    | 3d9310cce76642c49a10b61cdcdcb1cd                                            |
| creator               | 3d9310cce76642c49a10b61cdcdcb1cd:e4342cc6968c430ea7467623eb6b97ee           |
| ended_at              | None                                                                        |
| id                    | b6e4463b-463b-441d-9634-68f3d95b2ec8                                        |
| metrics               | snapshot.size: f30a5ca2-b47a-4a65-b1b0-fd3e912edec5                         |
|                       | volume.backup.size: 1e4f3ae3-8fd6-4241-8494-f5869dffe4b6                    |
|                       | volume.manage_existing.end: 1276a543-6505-46c9-8f62-6a201918ce6f            |
|                       | volume.manage_existing.start: ebb3e582-1f38-4035-9029-cf469e658fbe          |
|                       | volume.manage_existing_snapshot.end: 1b19c7f9-1ef3-46a7-80a6-d17558ba4add   |
|                       | volume.manage_existing_snapshot.start: d624c857-a4f4-421e-9b35-3a0a22ae1f0b |
|                       | volume.size: 74b0f14a-977b-4ad1-b646-1551b7d69865                           |
|                       | volume.snapshot.size: 7d766aa6-be35-4313-8077-15fb47e56f81                  |
|                       | volume: 76391447-8775-4985-ab36-9472d4cc5db5                                |
| original_resource_id  | b6e4463b-463b-441d-9634-68f3d95b2ec8                                        |
| project_id            | b0a43e2bb36e4910a6730e2d823ef08f                                            |
| revision_end          | None                                                                        |
| revision_start        | 2020-10-20T05:27:37.099628+00:00                                            |
| started_at            | 2020-10-20T05:27:37.099541+00:00                                            |
| type                  | volume                                                                      |
| user_id               | 2fc8ac9126654aaa865bd2951dce769b                                            |
+-----------------------+-----------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$

Comment 40 errata-xmlrpc 2020-10-28 18:31:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openstack-ceilometer bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4396