Bug 2232489 - Backing Store failed to reach OPTIMAL state
Summary: Backing Store failed to reach OPTIMAL state
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.13
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Nimrod Becker
QA Contact: krishnaram Karthick
URL:
Whiteboard:
: 2232490 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-17 06:03 UTC by Tiffany Nguyen
Modified: 2023-08-17 16:26 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)

Description Tiffany Nguyen 2023-08-17 06:03:22 UTC
Description of problem (please be detailed as possible and provide log
snippests):

PV-backingstore creation is failing to create:
 
ocs_ci.ocs.exceptions.TimeoutExpiredError: Timed Out: Backing Store pv-backingstore-3cc6ebeba6344c8da52fff3a never reached OPTIMAL state


From test logs backingstore creation didn't make it in time to OPTIMAL state. It ended in INITIALIZING state.

2023-08-11 14:02:51,216 - MainThread - INFO - ocs_ci.utility.utils.exec_cmd.619 - Executing command: oc get backingstore -n openshift-storage --kubeconfig /home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig pv-backingstore-3cc6ebeba6344c8da52fff3a -o=jsonpath=`{.status.mode.modeCode}`
2023-08-11 14:02:51,334 - MainThread - DEBUG - ocs_ci.utility.utils.exec_cmd.640 - Command stdout: `INITIALIZING`
2023-08-11 14:02:51,335 - MainThread - DEBUG - ocs_ci.utility.utils.exec_cmd.649 - Command stderr is empty
2023-08-11 14:02:51,335 - MainThread - DEBUG - ocs_ci.utility.utils.exec_cmd.650 - Command return code: 0
2023-08-11 14:02:51,335 - MainThread - INFO - ocs_ci.utility.utils.__iter__.1258 - Going to sleep for 15 seconds before next iteration
2023-08-11 14:02:53,433 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.421 - GET https://prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com/api/v1/query
2023-08-11 14:02:53,434 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.422 - headers={'Authorization': 'Bearer sha256~vU_KnKPErE8Is0h4Htoq6kcz2TeECXm-gSsMaT1NqTM'}
2023-08-11 14:02:53,434 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.423 - verify=False
2023-08-11 14:02:53,434 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.424 - params={'query': 'cluster:ceph_disk_latency:join_ceph_node_disk_irate1m', 'time': '1691762573.433877'}
2023-08-11 14:02:53,436 - Thread-2 - DEBUG - urllib3.connectionpool._new_conn.939 - Starting new HTTPS connection (1): prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com:443
2023-08-11 14:02:53,448 - Thread-2 - DEBUG - urllib3.connectionpool._make_request.433 - https://prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com:443 "GET /api/v1/query?query=cluster%3Aceph_disk_latency%3Ajoin_ceph_node_disk_irate1m&time=1691762573.433877 HTTP/1.1" 200 200
2023-08-11 14:02:53,451 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.validate_status.303 - content value: {'status': 'success', 'data': {'resultType': 'vector', 'result': [{'metric': {'__name__': 'cluster:ceph_disk_latency:join_ceph_node_disk_irate1m', 'namespace': 'openshift-storage'}, 'value': [1691762573.434, '0.0013252453008454552']}]}}
2023-08-11 14:02:58,456 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.421 - GET https://prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com/api/v1/query
2023-08-11 14:02:58,456 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.422 - headers={'Authorization': 'Bearer sha256~vU_KnKPErE8Is0h4Htoq6kcz2TeECXm-gSsMaT1NqTM'}
2023-08-11 14:02:58,456 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.423 - verify=False
2023-08-11 14:02:58,457 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.424 - params={'query': 'cluster:ceph_disk_latency:join_ceph_node_disk_irate1m', 'time': '1691762578.456533'}

Version of all relevant components (if applicable):

ODF 4.13.2


Can this issue reproducible?
Seen on automation regression execution logs

Can this issue reproduce from the UI?
None

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Execute test_bucket_creation
2. Observe the test results


Additional info:
Test log:  http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-004vu1cms33-ua/j-004vu1cms33-ua_20230811T043954/logs/ocs-ci-logs-1691748600/by_outcome/failed/tests/manage/mcg/test_bucket_creation.py/TestBucketCreation/test_bucket_creation-1-OC-PVPOOL/logs

Comment 2 Tiffany Nguyen 2023-08-17 16:26:17 UTC
*** Bug 2232490 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.