Bug 2296301

Summary: ceph_health_status with query range returns bad values, when the cluster is idle and ceph status is OK
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Daniel Osypenko <dosypenk>
Component: ceph-monitoringAssignee: arun kumar mohan <amohan>
Status: NEW --- QA Contact: Harish NV Rao <hnallurv>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.16CC: amohan, odf-bz-bot, skatiyar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Osypenko 2024-07-08 14:40:56 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
Across several deployment types we are not able to get ceph_health_status Ok when the cluster workload is idle.
This happens with "Multi storage cluster", with  "Compact mode and In Transit encryption"
Issue happens even though the Ceph health status is OK, when running ceph command
[2024-05-27T08:15:40.290Z] 04:15:39 - MainThread - tests.conftest - [32mINFO[0m  - Checking for Ceph Health OK 

Version of all relevant components (if applicable):
ODF v4.16.0-108
OCP 4.16.0-0.nightly-2024-05-23-173505

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes

Is there any workaround available to the best of your knowledge?
no

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
4 failures observed

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:
new types of deployment

Steps to Reproduce:
1. 
2.
3.


Actual results:
no ceph health OK metrics in ranged queries

Expected results:
ceph health OK metrics are available and positive when the ceph health is Ok


Additional info:

must gather logs http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-023vii1cs30-t1/j-023vii1cs30-t1_20240526T234857/logs/testcases_1716787108/j-023vii1cs30-t1/


2024-05-27 04:32:00,399 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.query_range.549 - Performing prometheus range query 'ceph_health_status' over a time range (1716797812.3947108, 1716798719.0427072)
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.428 - GET https://prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com/api/v1/query_range
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.429 - headers={'Authorization': 'Bearer sha256~rBfBZ2WDHco6kELkjFgUImkeOH7C4dZ0WuTyT3yqr_U'}
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.430 - verify=False
2024-05-27 04:32:00,400 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.431 - params={'query': 'ceph_health_status', 'start': 1716797812.3947108, 'end': 1716798719.0427072, 'step': 15}
2024-05-27 04:32:00,405 - MainThread - DEBUG - urllib3.connectionpool._new_conn.1019 - Starting new HTTPS connection (1): prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com:443
2024-05-27 04:32:00,457 - MainThread - DEBUG - urllib3.connectionpool._make_request.474 - https://prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com:443 "GET /api/v1/query_range?query=ceph_health_status&start=1716797812.3947108&end=1716798719.0427072&step=15 HTTP/1.1" 200 440
2024-05-27 04:32:00,495 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.validate_status.304 - content value: {'status': 'success', 'data': {'resultType': 'matrix', 'result': [{'metric': {'__name__': 'ceph_health_status', 'container': 'mgr', 'endpoint': 'http-metrics', 'instance': '10.129.0.102:9283', 'job': 'rook-ceph-mgr', 'managedBy': 'ocs-storagecluster', 'namespace': 'openshift-storage', 'pod': 'rook-ceph-mgr-a-697476d944-wvtm7', 'service': 'rook-ceph-mgr'}, 'values': [[1716797812.395, '0'], [1716797827.395, '0'], [1716797842.395, '0'], [1716797857.395, '0'], [1716797872.395, '1'], [1716797887.395, '1'], [1716797902.395, '0'], [1716797917.395, '0'], [1716797932.395, '0'], [1716797947.395, '0'], [1716797962.395, '1'], [1716797977.395, '1'], [1716797992.395, '0'], [1716798007.395, '0'], [1716798022.395, '0'], [1716798037.395, '0'], [1716798052.395, '0'], [1716798067.395, '0'], [1716798082.395, '0'], [1716798097.395, '0'], [1716798112.395, '0'], [1716798127.395, '0'], [1716798142.395, '0'], [1716798157.395, '0'], [1716798172.395, '0'], [1716798187.395, '0'], [1716798202.395, '0'], [1716798217.395, '0'], [1716798232.395, '0'], [1716798247.395, '0'], [1716798262.395, '0'], [1716798277.395, '0'], [1716798292.395, '0'], [1716798307.395, '0'], [1716798322.395, '0'], [1716798337.395, '0'], [1716798352.395, '0'], [1716798367.395, '0'], [1716798382.395, '0'], [1716798397.395, '0'], [1716798412.395, '0'], [1716798427.395, '0'], [1716798442.395, '0'], [1716798457.395, '0'], [1716798472.395, '0'], [1716798487.395, '0'], [1716798502.395, '0'], [1716798517.395, '0'], [1716798532.395, '0'], [1716798547.395, '0'], [1716798562.395, '0'], [1716798577.395, '0'], [1716798592.395, '0'], [1716798607.395, '0'], [1716798622.395, '0'], [1716798637.395, '0'], [1716798652.395, '0'], [1716798667.395, '0'], [1716798682.395, '0'], [1716798697.395, '0'], [1716798712.395, '0']]}]}}
2024-05-27 04:32:00,496 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.query_range.594 - there are no holes in the data
2024-05-27 04:32:00,499 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.117 - Validating a result of a range query
2024-05-27 04:32:00,500 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.136 - checking metric {'__name__': 'ceph_health_status', 'container': 'mgr', 'endpoint': 'http-metrics', 'instance': '10.129.0.102:9283', 'job': 'rook-ceph-mgr', 'managedBy': 'ocs-storagecluster', 'namespace': 'openshift-storage', 'pod': 'rook-ceph-mgr-a-697476d944-wvtm7', 'service': 'rook-ceph-mgr'}
2024-05-27 04:32:00,500 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.140 - metrics for ceph_health_status starts at 2024-05-27 08:16:52.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:16:52.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:07.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:22.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:37.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:17:52.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:18:07.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:22.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:37.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:52.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:19:07.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:19:22.395000
2024-05-27 04:32:00,502 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:19:37.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:19:52.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:07.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:22.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:37.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:52.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:07.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:22.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:37.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:52.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:07.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:22.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:37.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:52.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:07.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:22.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:37.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:52.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:07.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:22.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:37.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:52.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:07.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:22.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:37.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:52.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:07.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:22.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:37.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:52.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:07.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:22.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:37.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:52.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:07.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:22.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:37.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:52.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:07.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:22.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:37.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:52.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:07.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:22.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:37.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:52.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:07.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:22.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:37.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:52.395000
2024-05-27 04:32:00,509 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.172 - No invalid values detected
2024-05-27 04:32:00,509 - MainThread - ERROR - tests.functional.monitoring.prometheus.metrics.test_monitoring_defaults.test_monitoring_reporting_ok_when_idle.210 - ceph_health_status should report 0 (health ok) as expected

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/678/21547/1024366/1024404/log?logParams=history%3D1024404%26page.page%3D1