2296301 – ceph_health_status with query range returns bad values, when the cluster is idle and ceph status is OK

Bug 2296301 - ceph_health_status with query range returns bad values, when the cluster is idle and ceph status is OK

Summary: ceph_health_status with query range returns bad values, when the cluster is i...

Keywords:
Status:	NEW
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	ceph-monitoring
Sub Component:
Version:	4.16
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	arun kumar mohan
QA Contact:	Harish NV Rao
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2024-07-08 14:40 UTC by Daniel Osypenko
Modified:	2024-09-30 13:41 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OCSBZM-8626	0	None	None	None	2024-08-26 13:16:52 UTC

Description Daniel Osypenko 2024-07-08 14:40:56 UTC

Description of problem (please be detailed as possible and provide log
snippests):
Across several deployment types we are not able to get ceph_health_status Ok when the cluster workload is idle.
This happens with "Multi storage cluster", with  "Compact mode and In Transit encryption"
Issue happens even though the Ceph health status is OK, when running ceph command
[2024-05-27T08:15:40.290Z] 04:15:39 - MainThread - tests.conftest - [32mINFO[0m  - Checking for Ceph Health OK 

Version of all relevant components (if applicable):
ODF v4.16.0-108
OCP 4.16.0-0.nightly-2024-05-23-173505

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes

Is there any workaround available to the best of your knowledge?
no

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
4 failures observed

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:
new types of deployment

Steps to Reproduce:
1. 
2.
3.


Actual results:
no ceph health OK metrics in ranged queries

Expected results:
ceph health OK metrics are available and positive when the ceph health is Ok


Additional info:

must gather logs http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-023vii1cs30-t1/j-023vii1cs30-t1_20240526T234857/logs/testcases_1716787108/j-023vii1cs30-t1/


2024-05-27 04:32:00,399 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.query_range.549 - Performing prometheus range query 'ceph_health_status' over a time range (1716797812.3947108, 1716798719.0427072)
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.428 - GET https://prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com/api/v1/query_range
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.429 - headers={'Authorization': 'Bearer sha256~rBfBZ2WDHco6kELkjFgUImkeOH7C4dZ0WuTyT3yqr_U'}
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.430 - verify=False
2024-05-27 04:32:00,400 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.431 - params={'query': 'ceph_health_status', 'start': 1716797812.3947108, 'end': 1716798719.0427072, 'step': 15}
2024-05-27 04:32:00,405 - MainThread - DEBUG - urllib3.connectionpool._new_conn.1019 - Starting new HTTPS connection (1): prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com:443
2024-05-27 04:32:00,457 - MainThread - DEBUG - urllib3.connectionpool._make_request.474 - https://prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com:443 "GET /api/v1/query_range?query=ceph_health_status&start=1716797812.3947108&end=1716798719.0427072&step=15 HTTP/1.1" 200 440
2024-05-27 04:32:00,495 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.validate_status.304 - content value: {'status': 'success', 'data': {'resultType': 'matrix', 'result': [{'metric': {'__name__': 'ceph_health_status', 'container': 'mgr', 'endpoint': 'http-metrics', 'instance': '10.129.0.102:9283', 'job': 'rook-ceph-mgr', 'managedBy': 'ocs-storagecluster', 'namespace': 'openshift-storage', 'pod': 'rook-ceph-mgr-a-697476d944-wvtm7', 'service': 'rook-ceph-mgr'}, 'values': [[1716797812.395, '0'], [1716797827.395, '0'], [1716797842.395, '0'], [1716797857.395, '0'], [1716797872.395, '1'], [1716797887.395, '1'], [1716797902.395, '0'], [1716797917.395, '0'], [1716797932.395, '0'], [1716797947.395, '0'], [1716797962.395, '1'], [1716797977.395, '1'], [1716797992.395, '0'], [1716798007.395, '0'], [1716798022.395, '0'], [1716798037.395, '0'], [1716798052.395, '0'], [1716798067.395, '0'], [1716798082.395, '0'], [1716798097.395, '0'], [1716798112.395, '0'], [1716798127.395, '0'], [1716798142.395, '0'], [1716798157.395, '0'], [1716798172.395, '0'], [1716798187.395, '0'], [1716798202.395, '0'], [1716798217.395, '0'], [1716798232.395, '0'], [1716798247.395, '0'], [1716798262.395, '0'], [1716798277.395, '0'], [1716798292.395, '0'], [1716798307.395, '0'], [1716798322.395, '0'], [1716798337.395, '0'], [1716798352.395, '0'], [1716798367.395, '0'], [1716798382.395, '0'], [1716798397.395, '0'], [1716798412.395, '0'], [1716798427.395, '0'], [1716798442.395, '0'], [1716798457.395, '0'], [1716798472.395, '0'], [1716798487.395, '0'], [1716798502.395, '0'], [1716798517.395, '0'], [1716798532.395, '0'], [1716798547.395, '0'], [1716798562.395, '0'], [1716798577.395, '0'], [1716798592.395, '0'], [1716798607.395, '0'], [1716798622.395, '0'], [1716798637.395, '0'], [1716798652.395, '0'], [1716798667.395, '0'], [1716798682.395, '0'], [1716798697.395, '0'], [1716798712.395, '0']]}]}}
2024-05-27 04:32:00,496 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.query_range.594 - there are no holes in the data
2024-05-27 04:32:00,499 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.117 - Validating a result of a range query
2024-05-27 04:32:00,500 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.136 - checking metric {'__name__': 'ceph_health_status', 'container': 'mgr', 'endpoint': 'http-metrics', 'instance': '10.129.0.102:9283', 'job': 'rook-ceph-mgr', 'managedBy': 'ocs-storagecluster', 'namespace': 'openshift-storage', 'pod': 'rook-ceph-mgr-a-697476d944-wvtm7', 'service': 'rook-ceph-mgr'}
2024-05-27 04:32:00,500 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.140 - metrics for ceph_health_status starts at 2024-05-27 08:16:52.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:16:52.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:07.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:22.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:37.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:17:52.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:18:07.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:22.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:37.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:52.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:19:07.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:19:22.395000
2024-05-27 04:32:00,502 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:19:37.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:19:52.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:07.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:22.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:37.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:52.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:07.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:22.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:37.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:52.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:07.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:22.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:37.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:52.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:07.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:22.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:37.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:52.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:07.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:22.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:37.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:52.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:07.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:22.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:37.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:52.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:07.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:22.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:37.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:52.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:07.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:22.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:37.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:52.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:07.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:22.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:37.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:52.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:07.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:22.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:37.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:52.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:07.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:22.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:37.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:52.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:07.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:22.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:37.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:52.395000
2024-05-27 04:32:00,509 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.172 - No invalid values detected
2024-05-27 04:32:00,509 - MainThread - ERROR - tests.functional.monitoring.prometheus.metrics.test_monitoring_defaults.test_monitoring_reporting_ok_when_idle.210 - ceph_health_status should report 0 (health ok) as expected

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/678/21547/1024366/1024404/log?logParams=history%3D1024404%26page.page%3D1

Note You need to log in before you can comment on or make changes to this bug.