Bug 2232490
| Summary: | Backing Store failed to reach OPTIMAL state | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Tiffany Nguyen <tunguyen> |
| Component: | Multi-Cloud Object Gateway | Assignee: | Nimrod Becker <nbecker> |
| Status: | CLOSED DUPLICATE | QA Contact: | krishnaram Karthick <kramdoss> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.13 | CC: | odf-bz-bot |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-08-17 16:26:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 2232489 *** |
Description of problem (please be detailed as possible and provide log snippests): PV-backingstore creation is failing to create: ocs_ci.ocs.exceptions.TimeoutExpiredError: Timed Out: Backing Store pv-backingstore-3cc6ebeba6344c8da52fff3a never reached OPTIMAL state From test logs backingstore creation didn't make it in time to OPTIMAL state. It ended in INITIALIZING state. 2023-08-11 14:02:51,216 - MainThread - INFO - ocs_ci.utility.utils.exec_cmd.619 - Executing command: oc get backingstore -n openshift-storage --kubeconfig /home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig pv-backingstore-3cc6ebeba6344c8da52fff3a -o=jsonpath=`{.status.mode.modeCode}` 2023-08-11 14:02:51,334 - MainThread - DEBUG - ocs_ci.utility.utils.exec_cmd.640 - Command stdout: `INITIALIZING` 2023-08-11 14:02:51,335 - MainThread - DEBUG - ocs_ci.utility.utils.exec_cmd.649 - Command stderr is empty 2023-08-11 14:02:51,335 - MainThread - DEBUG - ocs_ci.utility.utils.exec_cmd.650 - Command return code: 0 2023-08-11 14:02:51,335 - MainThread - INFO - ocs_ci.utility.utils.__iter__.1258 - Going to sleep for 15 seconds before next iteration 2023-08-11 14:02:53,433 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.421 - GET https://prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com/api/v1/query 2023-08-11 14:02:53,434 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.422 - headers={'Authorization': 'Bearer sha256~vU_KnKPErE8Is0h4Htoq6kcz2TeECXm-gSsMaT1NqTM'} 2023-08-11 14:02:53,434 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.423 - verify=False 2023-08-11 14:02:53,434 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.424 - params={'query': 'cluster:ceph_disk_latency:join_ceph_node_disk_irate1m', 'time': '1691762573.433877'} 2023-08-11 14:02:53,436 - Thread-2 - DEBUG - urllib3.connectionpool._new_conn.939 - Starting new HTTPS connection (1): prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com:443 2023-08-11 14:02:53,448 - Thread-2 - DEBUG - urllib3.connectionpool._make_request.433 - https://prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com:443 "GET /api/v1/query?query=cluster%3Aceph_disk_latency%3Ajoin_ceph_node_disk_irate1m&time=1691762573.433877 HTTP/1.1" 200 200 2023-08-11 14:02:53,451 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.validate_status.303 - content value: {'status': 'success', 'data': {'resultType': 'vector', 'result': [{'metric': {'__name__': 'cluster:ceph_disk_latency:join_ceph_node_disk_irate1m', 'namespace': 'openshift-storage'}, 'value': [1691762573.434, '0.0013252453008454552']}]}} 2023-08-11 14:02:58,456 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.421 - GET https://prometheus-k8s-openshift-monitoring.apps.j-004vu1cms33-ua.qe.rh-ocs.com/api/v1/query 2023-08-11 14:02:58,456 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.422 - headers={'Authorization': 'Bearer sha256~vU_KnKPErE8Is0h4Htoq6kcz2TeECXm-gSsMaT1NqTM'} 2023-08-11 14:02:58,456 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.423 - verify=False 2023-08-11 14:02:58,457 - Thread-2 - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.424 - params={'query': 'cluster:ceph_disk_latency:join_ceph_node_disk_irate1m', 'time': '1691762578.456533'} Version of all relevant components (if applicable): ODF 4.13.2 Can this issue reproducible? Seen on automation regression execution logs Can this issue reproduce from the UI? None If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Execute test_bucket_creation 2. Observe the test results Additional info: Test log: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-004vu1cms33-ua/j-004vu1cms33-ua_20230811T043954/logs/ocs-ci-logs-1691748600/by_outcome/failed/tests/manage/mcg/test_bucket_creation.py/TestBucketCreation/test_bucket_creation-1-OC-PVPOOL/logs