Bug 2250586

Description of problem:

When we try to delete a metric using the following command, it fails due to the following HTTP 406 error.

~~~
(overcloud) [stack@undercloud mytempest]$ openstack metric delete 23dce950-1f1c-4a65-8dc1-54d9639dac4d --debug
  :
REQ: curl -g -i -X DELETE http://10.0.0.188:8041/v1/metric/23dce950-1f1c-4a65-8dc1-54d9639dac4d -H "Accept: application/json" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.4.0 python-requests/2.25.1 CPython/3.9.16" -H "X-Auth-Token: {SHA256}92d9220bc98b20ee605e383bdc5a1d795ceb54804c215ecda5c110a7afd2b642"
Starting new HTTP connection (1): 10.0.0.188:8041
http://10.0.0.188:8041 "DELETE /v1/metric/23dce950-1f1c-4a65-8dc1-54d9639dac4d HTTP/1.1" 406 61
RESP: [406] content-length: 61 content-type: application/json date: Mon, 20 Nov 2023 04:41:48 GMT server: Apache
RESP BODY: {"code": 406, "title": "Not Acceptable", "description": null}
Not Acceptable (HTTP 406)
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/metric_cli.py", line 134, in take_action
    utils.get_client(self).metric.delete(
  File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/metric.py", line 190, in delete
    self._delete(url)
  File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/base.py", line 53, in _delete
    return self.client.api.delete(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 410, in delete
    return self.request(url, 'DELETE', **kwargs)
  File "/usr/lib/python3.9/site-packages/gnocchiclient/client.py", line 52, in request
    raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.NotAcceptable: Not Acceptable (HTTP 406)
clean_up CliMetricDelete: Not Acceptable (HTTP 406)
END return value: 1
~~~

When gnocchi client sends a request, Accept header will be "application/json".
So gnocchi API must support "application/json".

~~~
https://github.com/gnocchixyz/python-gnocchiclient/blob/master/gnocchiclient/v1/base.py#L20-L22
class Manager(object):
    DEFAULT_HEADERS = {
        "Accept": "application/json",
    }
~~~

The reason why this error happens is that the current gnocchi package is missing the following change for some reason.
That's why gnocchi API doesn't support "application/json" and 406 error occurs.

https://github.com/gnocchixyz/gnocchi/commit/ba5d73d5009c5fe5b71053546a240883a9719719#diff-731d47f3817508cb3d32ab7d0f8287c2f228c64894006f7a6b2ddae2f7a62e7aL530-R530

~~~
(overcloud) [stack@undercloud rpm]$ dnf download python3-gnocchi-4.4.3-1.20230209190902.60c2b56.el9ost.noarch
(overcloud) [stack@undercloud rpm]$ rpm2cipo gnocchi-api-4.4.3-1.20230209190902.60c2b56.el9ost.noarch.rpm| cpio -idv
(overcloud) [stack@undercloud rpm]$ vi usr/lib/python3.9/site-packages/gnocchi/rest/api.py
  :
class MetricController(rest.RestController):
  :
    @pecan.expose() <==============================================(*) The current implementation is missing 'json' here.
    def delete(self):
        self.enforce_metric("delete metric")
        try:
            pecan.request.indexer.delete_metric(self.metric.id)
        except indexer.NoSuchMetric as e:
            abort(404, six.text_type(e))
  :
~~~

If I make the following change(*) manually, the metric deletion succeeds.

~~~
[root@overcloud-controller-2 ~]# podman exec -it gnocchi_api bash 
[root@overcloud-controller-2 /]# vi /usr/lib/python3.9/site-packages/gnocchi/rest/api.py 

       :
class MetricController(rest.RestController):
       :
    @pecan.expose('json')  <====================================(*)
    def delete(self):
        self.enforce_metric("delete metric")
        try:
            pecan.request.indexer.delete_metric(self.metric.id)
        except indexer.NoSuchMetric as e:
            abort(404, six.text_type(e))
       :

[root@overcloud-controller-2 /]# exit
exit
[root@overcloud-controller-2 ~]# systemctl restart tripleo_gnocchi_api

  ==========> Manually added 'json' on controller-2


(overcloud) [stack@undercloud mytempest]$ openstack metric delete 23dce950-1f1c-4a65-8dc1-54d9639dac4d ==> fails
(overcloud) [stack@undercloud mytempest]$ openstack metric delete 23dce950-1f1c-4a65-8dc1-54d9639dac4d ==> fails
(overcloud) [stack@undercloud mytempest]$ openstack metric delete 23dce950-1f1c-4a65-8dc1-54d9639dac4d ==> succeeds

(overcloud) [stack@undercloud mytempest]$ ansible -i ~/overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml Controller -b -m shell -a 'grep -r "/v1/metric/23dce950-1f1c-4a65-8dc1-54d9639dac4d" /var/log/containers/haproxy -h'
  :
overcloud-controller-1 | CHANGED | rc=0 >>
  :
Nov 20 04:41:44 overcloud-controller-1 haproxy[7]: 10.0.0.254:59872 [20/Nov/2023:04:41:44.358] gnocchi gnocchi/overcloud-controller-0.internalapi.example.com 0/0/0/66/66 406 196 - - ---- 2/1/0/0/0 0/0 "DELETE /v1/metric/23dce950-1f1c-4a65-8dc1-54d9639dac4d HTTP/1.1"
Nov 20 04:41:49 overcloud-controller-1 haproxy[7]: 10.0.0.254:50460 [20/Nov/2023:04:41:48.951] gnocchi gnocchi/overcloud-controller-1.internalapi.example.com 0/0/0/112/112 406 196 - - ---- 2/1/0/0/0 0/0 "DELETE /v1/metric/23dce950-1f1c-4a65-8dc1-54d9639dac4d HTTP/1.1"
Nov 20 04:41:52 overcloud-controller-1 haproxy[7]: 10.0.0.254:50462 [20/Nov/2023:04:41:52.218] gnocchi gnocchi/overcloud-controller-2.internalapi.example.com 0/0/0/481/481 204 87 - - ---- 2/1/0/0/0 0/0 "DELETE /v1/metric/23dce950-1f1c-4a65-8dc1-54d9639dac4d HTTP/1.1"
  :

  ===========> When the delete api call is redirected to controller-2 by HAProxy, it succeeds.
~~~

As a workaround, we can call the API with "Accept: text/html" instead of "Accept: application/json" manually.

~~~
(overcloud) [stack@undercloud mytempest]$ curl -g -i -X DELETE http://10.0.0.188:8041/v1/metric/0ae0700c-85d8-4383-9f8d-983cc8b59d18 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.4.0 python-requests/2.25.1 CPython/3.9.16" -H "X-Auth-Token: gAAAAABlWt4h3TEcy7cO6HpGwaM_16kRXw5zdT46bq4a-yDw8EyXiw3HkU9IO4B6iWBnCkX37ghCSdzwEFN-GrDXFz_s3iqr-BxWAmQzY3mskvSI_ZezxPRMcZAnYQZWYCaBcTuxOTfvBAiqrNPhwkm_CTBpfON5oTzFEs8XUTX8m4vckCDRNWg"
HTTP/1.1 406 Not Acceptable
date: Mon, 20 Nov 2023 04:18:50 GMT
server: Apache
content-length: 61
content-type: application/json

{"code": 406, "title": "Not Acceptable", "description": null}(overcloud) [stack@undercloud mytempest]$ 

  ===> Fail

(overcloud) [stack@undercloud mytempest]$ curl -g -i -X DELETE http://10.0.0.188:8041/v1/metric/0ae0700c-85d8-4383-9f8d-983cc8b59d18 -H "Accept: text/html" -H "User-Agent: openstacksdk/0.55.1 keystoneauth1/4.4.0 python-requests/2.25.1 CPython/3.9.16" -H "X-Auth-Token: gAAAAABlWt4h3TEcy7cO6HpGwaM_16kRXw5zdT46bq4a-yDw8EyXiw3HkU9IO4B6iWBnCkX37ghCSdzwEFN-GrDXFz_s3iqr-BxWAmQzY3mskvSI_ZezxPRMcZAnYQZWYCaBcTuxOTfvBAiqrNPhwkm_CTBpfON5oTzFEs8XUTX8m4vckCDRNWg"
HTTP/1.1 204 No Content
date: Mon, 20 Nov 2023 04:19:03 GMT
server: Apache

  ===> Succeed
~~~


Version-Release number of selected component (if applicable):
RHOSP 17.1

How reproducible:
Steps to Reproduce:
1. Deploy overcloud with gnocchi
2. Create a metric
3. Delete the metric

Actual results:
We cannot delete a metric using `openstack metric delete` command due to 406 error


Expected results:
We can delete a metric using `openstack metric delete` command without errors


Additional info:
My customer hit this error on the following tempest test:

~~~
(overcloud) [stack@undercloud mytempest]$ tempest run --regex heat_tempest_plugin.tests.scenario.test_aodh_alarm.AodhAlarmTest.test_alarm

{0} heat_tempest_plugin.tests.scenario.test_aodh_alarm.AodhAlarmTest.test_alarm [65.359775s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/heat_tempest_plugin/tests/scenario/test_aodh_alarm.py", line 67, in test_alarm
        self.metric_client.metric.delete(metric['id'])
      File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/metric.py", line 190, in delete
        self._delete(url)
      File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/base.py", line 53, in _delete
        return self.client.api.delete(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 410, in delete
        return self.request(url, 'DELETE', **kwargs)
      File "/usr/lib/python3.9/site-packages/gnocchiclient/client.py", line 52, in request
        raise exceptions.from_response(resp, method)
    gnocchiclient.exceptions.NotAcceptable: Not Acceptable (HTTP 406)
~~~