+++ This bug was initially created as a clone of Bug #2265574 +++

+++ This bug was initially created as a clone of Bug #2251055 +++

Current Metrics sent by the ceph-exporter don't include user or bucket on the metric name, for example:

ceph_rgw_op_del_obj_bytes{Bucket="bkt", instance="localhost:9926", instance_id="8000", job="radosgw"}
ceph_rgw_op_del_obj_bytes{User="anonymous", instance="localhost:9926", instance_id="8000", job="radosgw"}

This adds an extra dimension for Prometheus and Grafana to query and filter out the specific data, making it very complex and resource-expensive to configure Grafana dashboards.

The request is to modify the ceph-exporter so it constructs the metrics it sends out to Prometheus using the metric's key. In the future, they could be for anything else, like groups, accounts,etc.

Examples on what the final metric being sent to Prometheus should look like:

-- GLOBAL --
ceph_rgw_op_del_obj_bytes
ceph_rgw_op_del_obj_bytes{instance="localhost:9926", instance_id="8000", job="radosgw"}


-- Bucket operations --
ceph_rgw_op_del_bucket_obj_bytes

ceph_rgw_op_del_bucket_obj_bytes{Bucket="bkt", instance="localhost:9926", instance_id="8000", job="radosgw"}
ceph_rgw_op_del_bucket_obj_bytes{Bucket="bkt2", instance="localhost:9926", instance_id="8000", job="radosgw"}

-- User operations --
ceph_rgw_op_del_user_obj_bytes

ceph_rgw_op_del_user_obj_bytes{User="anonymous", instance="localhost:9926", instance_id="8000", job="radosgw"}
ceph_rgw_op_del_user_obj_bytes{User="test3", instance="localhost:9926", instance_id="8000", job="radosgw"}

--- Additional comment from Ali Maredia on 2024-02-22 19:19:26 UTC ---

The solution for this issue has been merged upstream at https://github.com/ceph/ceph/pull/54623 (only the first commit) and that commit has been pushed downstream.

--- Additional comment from  on 2024-02-22 22:10:05 UTC ---

Builds are ready for testing. We need a qa_ack+ in order to attach this BZ to the errata advisory and move to ON_QA.

--- Additional comment from Madhavi Kasturi on 2024-02-26 05:51:06 UTC ---

Hi Ali,

Could you please help with a reproducer to verify this BZ.

Thanks,
Madhavi

--- Additional comment from errata-xmlrpc on 2024-02-26 06:29:19 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:126567-01
https://errata.engineering.redhat.com/advisory/126567

--- Additional comment from errata-xmlrpc on 2024-02-26 06:29:27 UTC ---

This bug has been added to advisory RHBA-2024:126567 by Thomas Serlin (tserlin)

--- Additional comment from Chaithra on 2024-03-04 07:07:06 UTC ---

Hi Ali,
Have verified metrics on: ceph version 18.2.1-40.el9cp

New format that seen is as below:

##########################################################################

# HELP ceph_rgw_op_get_obj_ops Gets
# TYPE ceph_rgw_op_get_obj_ops counter
ceph_rgw_op_get_obj_ops{instance_id="ceph-ck-mp-q74oz9-node5"} 9


# HELP ceph_rgw_op_per_bucket_get_obj_ops Gets
# TYPE ceph_rgw_op_per_bucket_get_obj_ops counter
ceph_rgw_op_per_bucket_get_obj_ops{bucket="aws-bkt1",instance_id="ceph-ck-mp-q74oz9-node5"} 2

# HELP ceph_rgw_op_per_user_get_obj_ops Gets
# TYPE ceph_rgw_op_per_user_get_obj_ops counter
ceph_rgw_op_per_user_get_obj_ops{instance_id="ceph-ck-mp-q74oz9-node5",user="ckulal"} 9

Please let me know if this observation suffices the BZ Verification.

Thanks and Regards,
Chaithra

--- Additional comment from Ali Maredia on 2024-03-07 08:05:11 UTC ---

This looks good to me. Thanks Chaithra!

- Ali

--- Additional comment from Chaithra on 2024-03-07 09:13:41 UTC ---

Thanks Ali, Moving BZ to Verified.

--- Additional comment from Akash Raj on 2024-03-14 05:15:15 UTC ---

Hi Ali

Please confirm if the doc text for this BZ needs to be added in the release notes. If yes, please provide the doc type and text.

Thanks.