Bug 2123697
| Summary: | report-status-to-provider pod in CreateContainerError state on consumer | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Jilju Joy <jijoy> | |
| Component: | build | Assignee: | Tamil <tmuthami> | |
| Status: | CLOSED ERRATA | QA Contact: | Jilju Joy <jijoy> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.11 | CC: | aeyal, dbindra, dkhandel, kramdoss, madam, muagarwa, ocs-bugs, odf-bz-bot, sheggodu | |
| Target Milestone: | --- | |||
| Target Release: | ODF 4.11.1 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2123724 (view as bug list) | Environment: | ||
| Last Closed: | 2022-09-14 15:15:05 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2120314, 2123724 | |||
After debugging Jilju's cluster, I got to know that status-reporter binary is missing from the container which resulted into status-reporter pod going into the CreateContainerError state. I have informed the build team about it and they have raised the MR to fix the build pipeline. MR: https://gitlab.cee.redhat.com/ceph/rhodf/-/merge_requests/689 proposing the fix into 4.11.1 as this blocks verification of https://bugzilla.redhat.com/show_bug.cgi?id=2120314#c11 *** Bug 2123724 has been marked as a duplicate of this bug. *** Verified in version: ODF 4.11.1-8 OCP 4.10.30 ocs-osd-deployer.v2.0.5 Pods "report-status-to-provider" are in Completed state in the consumer cluster. $ oc get pods -o wide | grep report-status-to-provider report-status-to-provider-27716276-wn6fw 0/1 Completed 0 2m57s 10.129.2.49 ip-10-0-173-22.ec2.internal <none> <none> report-status-to-provider-27716277-nrn4z 0/1 Completed 0 117s 10.129.2.50 ip-10-0-173-22.ec2.internal <none> <none> report-status-to-provider-27716278-dvcpd 0/1 Completed 0 57s 10.131.0.46 ip-10-0-157-29.ec2.internal <none> <none> Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Data Foundation 4.11.1 Bug Fix Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:6525 |
Description of problem: Many instances of the pod "report-status-to-provider" are remaining in the state "CreateContainerError" in the consumer cluster. $ oc get pods NAME READY STATUS RESTARTS AGE 4747b56e58799d351945d54657efb0b83a8ca1d00dd7e5380d06d3fa794tw4q 0/1 Completed 0 4h16m 6950ec9eb29172a76d40e4b2ff5e2376b711b37788e0686023b978d02f7xd9h 0/1 Completed 0 4h16m a4d77567539501a711bc141b9ebab2e741b78999a50cb7cc1b157c395czh584 0/1 Completed 0 4h16m addon-ocs-consumer-qe-catalog-7z42p 1/1 Running 0 4h13m alertmanager-managed-ocs-alertmanager-0 2/2 Running 0 4h10m alertmanager-managed-ocs-alertmanager-1 2/2 Running 0 4h10m alertmanager-managed-ocs-alertmanager-2 2/2 Running 0 4h10m b82aad7f30c569c3b7e910a94ee8198c9b387508021164ca4ac419b54f226pq 0/1 Completed 0 4h16m b8bc02ee61adb95617b7822454117ffec980e791abef5220e3c66ee11bv86ml 0/1 Completed 0 4h16m csi-addons-controller-manager-8455c9946d-8fq69 2/2 Running 0 56m csi-cephfsplugin-64xhk 3/3 Running 0 56m csi-cephfsplugin-dxws8 3/3 Running 0 56m csi-cephfsplugin-provisioner-5bb48774f9-2l9pb 6/6 Running 0 56m csi-cephfsplugin-provisioner-5bb48774f9-gwln2 6/6 Running 0 56m csi-cephfsplugin-rdzjh 3/3 Running 0 56m csi-rbdplugin-2d4mc 4/4 Running 0 56m csi-rbdplugin-688nf 4/4 Running 0 56m csi-rbdplugin-7ms8d 4/4 Running 0 56m csi-rbdplugin-provisioner-6987d55d87-57tfd 7/7 Running 0 56m csi-rbdplugin-provisioner-6987d55d87-wqcqn 7/7 Running 0 56m df4206ef9f6068058722977a3e2821d93212a70782e6a82a8a9e796903wdgzw 0/1 Completed 0 4h16m ocs-metrics-exporter-76bd86987c-jtpns 1/1 Running 0 56m ocs-operator-684f88fcc7-79sqx 1/1 Running 0 57m ocs-osd-controller-manager-545b5759db-glpk2 3/3 Running 0 78m odf-console-6c77d5df7f-zssq7 1/1 Running 0 57m odf-operator-controller-manager-7c8fb545df-4p775 2/2 Running 0 57m prometheus-managed-ocs-prometheus-0 3/3 Running 0 4h10m prometheus-operator-8547cc9f89-2v4l2 1/1 Running 0 4h15m redhat-operators-s6x69 1/1 Running 0 4h13m report-status-to-provider-27701857-fzvsl 0/1 CreateContainerError 0 56m report-status-to-provider-27701858-vgcq7 0/1 CreateContainerError 0 55m report-status-to-provider-27701859-xnwc9 0/1 CreateContainerError 0 54m report-status-to-provider-27701860-4m6nh 0/1 CreateContainerError 0 53m report-status-to-provider-27701861-srmjj 0/1 CreateContainerError 0 52m report-status-to-provider-27701862-8n7ds 0/1 CreateContainerError 0 51m report-status-to-provider-27701863-xmbvn 0/1 CreateContainerError 0 50m report-status-to-provider-27701864-sfx7c 0/1 CreateContainerError 0 49m report-status-to-provider-27701865-fccd5 0/1 CreateContainerError 0 48m report-status-to-provider-27701866-spn5h 0/1 CreateContainerError 0 47m report-status-to-provider-27701867-4rrcz 0/1 CreateContainerError 0 46m report-status-to-provider-27701868-5h4z8 0/1 CreateContainerError 0 45m report-status-to-provider-27701869-5shhf 0/1 CreateContainerError 0 44m report-status-to-provider-27701870-8sxr8 0/1 CreateContainerError 0 43m report-status-to-provider-27701871-22rjf 0/1 CreateContainerError 0 42m report-status-to-provider-27701872-fgx6n 0/1 CreateContainerError 0 41m report-status-to-provider-27701873-vgf7k 0/1 CreateContainerError 0 40m report-status-to-provider-27701874-pdllq 0/1 CreateContainerError 0 39m report-status-to-provider-27701875-ddjwh 0/1 CreateContainerError 0 38m report-status-to-provider-27701876-mk5wh 0/1 CreateContainerError 0 37m report-status-to-provider-27701877-5twxv 0/1 CreateContainerError 0 36m report-status-to-provider-27701878-x7ggm 0/1 CreateContainerError 0 35m report-status-to-provider-27701879-7s7zf 0/1 CreateContainerError 0 34m report-status-to-provider-27701880-6gv96 0/1 CreateContainerError 0 33m report-status-to-provider-27701881-hrszq 0/1 CreateContainerError 0 32m report-status-to-provider-27701882-9bpgp 0/1 CreateContainerError 0 31m report-status-to-provider-27701883-dwmv4 0/1 CreateContainerError 0 30m report-status-to-provider-27701884-gv7ps 0/1 CreateContainerError 0 29m report-status-to-provider-27701885-j4p7p 0/1 CreateContainerError 0 28m report-status-to-provider-27701886-7dmv8 0/1 CreateContainerError 0 27m report-status-to-provider-27701887-dmcjb 0/1 CreateContainerError 0 26m report-status-to-provider-27701888-pxm42 0/1 CreateContainerError 0 25m report-status-to-provider-27701889-g4fc6 0/1 CreateContainerError 0 24m report-status-to-provider-27701890-t9hql 0/1 CreateContainerError 0 23m report-status-to-provider-27701891-lzcts 0/1 CreateContainerError 0 22m report-status-to-provider-27701892-x8qmc 0/1 CreateContainerError 0 21m report-status-to-provider-27701893-bjwwn 0/1 CreateContainerError 0 20m report-status-to-provider-27701894-k4vsh 0/1 CreateContainerError 0 19m report-status-to-provider-27701895-gvmq6 0/1 CreateContainerError 0 18m report-status-to-provider-27701896-qf7nh 0/1 CreateContainerError 0 17m report-status-to-provider-27701897-mwlhz 0/1 CreateContainerError 0 16m report-status-to-provider-27701898-v7qbn 0/1 CreateContainerError 0 15m report-status-to-provider-27701899-vcbfs 0/1 CreateContainerError 0 14m report-status-to-provider-27701900-c55dr 0/1 CreateContainerError 0 13m report-status-to-provider-27701901-4snlz 0/1 CreateContainerError 0 12m report-status-to-provider-27701902-t5n7l 0/1 CreateContainerError 0 11m report-status-to-provider-27701903-znhrd 0/1 CreateContainerError 0 10m report-status-to-provider-27701904-99djk 0/1 CreateContainerError 0 9m32s report-status-to-provider-27701905-97bw7 0/1 CreateContainerError 0 8m32s report-status-to-provider-27701906-6bcg4 0/1 CreateContainerError 0 7m32s report-status-to-provider-27701907-hgbxl 0/1 CreateContainerError 0 6m32s report-status-to-provider-27701908-spwzp 0/1 CreateContainerError 0 5m32s report-status-to-provider-27701909-dfd8x 0/1 CreateContainerError 0 4m32s report-status-to-provider-27701910-rrfmx 0/1 CreateContainerError 0 3m32s report-status-to-provider-27701911-mv5pn 0/1 CreateContainerError 0 2m32s report-status-to-provider-27701912-wskz6 0/1 CreateContainerError 0 92s report-status-to-provider-27701913-lcpfm 0/1 CreateContainerError 0 32s rook-ceph-operator-85dc6665f-jrxpg 1/1 Running 2 (56m ago) 57m rook-ceph-tools-74cff48f58-b8k2n 1/1 Running 0 56m ODF 4.11.1 was installed after uninstalling ODF 4.10.5 in both consumer and provider cluster. This was done because the upgrade to ODF 4.11.1 was not starting automatically(followed the same steps we were using to upgrade to ODF 4.11.0 from 4.10). $ oc get managedocs -o yaml apiVersion: v1 items: - apiVersion: ocs.openshift.io/v1alpha1 kind: ManagedOCS metadata: creationTimestamp: "2022-09-02T06:18:53Z" finalizers: - managedocs.ocs.openshift.io generation: 1 name: managedocs namespace: openshift-storage resourceVersion: "452444" uid: 87bd030a-d780-4c60-adae-f004cc9ea421 spec: {} status: components: alertmanager: state: Ready prometheus: state: Ready storageCluster: state: Ready reconcileStrategy: strict kind: List metadata: resourceVersion: "" selfLink: "" consumer must-gather - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-sep2-c1/jijoy-sep2-c1_20220902T054021/logs/testcases_1662114794/ provider must-gather - http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jijoy-sep2-pr/jijoy-sep2-pr_20220902T041842/logs/testcases_1662115760/ ================================================================= Version-Release number of selected component (if applicable): OCP 4.10.28 ODF 4.11.1-5 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.11.1 NooBaa Operator 4.11.1 mcg-operator.v4.11.0 Succeeded ocs-operator.v4.11.1 OpenShift Container Storage 4.11.1 ocs-operator.v4.11.0 Succeeded ocs-osd-deployer.v2.0.5 OCS OSD Deployer 2.0.5 ocs-osd-deployer.v2.0.4 Succeeded odf-csi-addons-operator.v4.11.1 CSI Addons 4.11.1 odf-csi-addons-operator.v4.11.0 Succeeded odf-operator.v4.11.1 OpenShift Data Foundation 4.11.1 odf-operator.v4.11.0 Succeeded ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 ose-prometheus-operator.4.8.0 Succeeded route-monitor-operator.v0.1.422-151be96 Route Monitor Operator 0.1.422-151be96 route-monitor-operator.v0.1.420-b65f47e Succeeded ========================================================== How reproducible: Reporting the first occurrence Steps to Reproduce: 1. Upgrade provider and consumer from ODF 4.10.z to 4.11.1 (performed uninstallation of ODF 4.10.5 and installed ODF 4.11.1 to test this because upgrade process was not working) 2. Check the status of report-status-to-provider pod in the consumer. Actual results: $ oc get pods | grep report-status-to-provider report-status-to-provider-27701857-fzvsl 0/1 CreateContainerError 0 75m report-status-to-provider-27701858-vgcq7 0/1 CreateContainerError 0 74m report-status-to-provider-27701859-xnwc9 0/1 CreateContainerError 0 73m report-status-to-provider-27701860-4m6nh 0/1 CreateContainerError 0 72m report-status-to-provider-27701861-srmjj 0/1 CreateContainerError 0 71m report-status-to-provider-27701862-8n7ds 0/1 CreateContainerError 0 70m report-status-to-provider-27701863-xmbvn 0/1 CreateContainerError 0 69m report-status-to-provider-27701864-sfx7c 0/1 CreateContainerError 0 68m report-status-to-provider-27701865-fccd5 0/1 CreateContainerError 0 67m report-status-to-provider-27701866-spn5h 0/1 CreateContainerError 0 66m report-status-to-provider-27701867-4rrcz 0/1 CreateContainerError 0 65m report-status-to-provider-27701868-5h4z8 0/1 CreateContainerError 0 64m report-status-to-provider-27701869-5shhf 0/1 CreateContainerError 0 63m report-status-to-provider-27701870-8sxr8 0/1 CreateContainerError 0 62m report-status-to-provider-27701871-22rjf 0/1 CreateContainerError 0 61m report-status-to-provider-27701872-fgx6n 0/1 CreateContainerError 0 60m report-status-to-provider-27701873-vgf7k 0/1 CreateContainerError 0 59m report-status-to-provider-27701874-pdllq 0/1 CreateContainerError 0 58m report-status-to-provider-27701875-ddjwh 0/1 CreateContainerError 0 57m report-status-to-provider-27701876-mk5wh 0/1 CreateContainerError 0 56m report-status-to-provider-27701877-5twxv 0/1 CreateContainerError 0 55m report-status-to-provider-27701878-x7ggm 0/1 CreateContainerError 0 54m report-status-to-provider-27701879-7s7zf 0/1 CreateContainerError 0 53m report-status-to-provider-27701880-6gv96 0/1 CreateContainerError 0 52m report-status-to-provider-27701881-hrszq 0/1 CreateContainerError 0 51m report-status-to-provider-27701882-9bpgp 0/1 CreateContainerError 0 50m report-status-to-provider-27701883-dwmv4 0/1 CreateContainerError 0 49m report-status-to-provider-27701884-gv7ps 0/1 CreateContainerError 0 48m report-status-to-provider-27701885-j4p7p 0/1 CreateContainerError 0 47m report-status-to-provider-27701886-7dmv8 0/1 CreateContainerError 0 46m report-status-to-provider-27701887-dmcjb 0/1 CreateContainerError 0 45m report-status-to-provider-27701888-pxm42 0/1 CreateContainerError 0 44m report-status-to-provider-27701889-g4fc6 0/1 CreateContainerError 0 43m report-status-to-provider-27701890-t9hql 0/1 CreateContainerError 0 42m report-status-to-provider-27701891-lzcts 0/1 CreateContainerError 0 41m report-status-to-provider-27701892-x8qmc 0/1 CreateContainerError 0 40m report-status-to-provider-27701893-bjwwn 0/1 CreateContainerError 0 39m report-status-to-provider-27701894-k4vsh 0/1 CreateContainerError 0 38m report-status-to-provider-27701895-gvmq6 0/1 CreateContainerError 0 37m report-status-to-provider-27701896-qf7nh 0/1 CreateContainerError 0 36m report-status-to-provider-27701897-mwlhz 0/1 CreateContainerError 0 35m report-status-to-provider-27701898-v7qbn 0/1 CreateContainerError 0 34m report-status-to-provider-27701899-vcbfs 0/1 CreateContainerError 0 33m report-status-to-provider-27701900-c55dr 0/1 CreateContainerError 0 32m report-status-to-provider-27701901-4snlz 0/1 CreateContainerError 0 31m report-status-to-provider-27701902-t5n7l 0/1 CreateContainerError 0 30m report-status-to-provider-27701903-znhrd 0/1 CreateContainerError 0 29m report-status-to-provider-27701904-99djk 0/1 CreateContainerError 0 28m report-status-to-provider-27701905-97bw7 0/1 CreateContainerError 0 27m report-status-to-provider-27701906-6bcg4 0/1 CreateContainerError 0 26m report-status-to-provider-27701907-hgbxl 0/1 CreateContainerError 0 25m report-status-to-provider-27701908-spwzp 0/1 CreateContainerError 0 24m report-status-to-provider-27701909-dfd8x 0/1 CreateContainerError 0 23m report-status-to-provider-27701910-rrfmx 0/1 CreateContainerError 0 22m report-status-to-provider-27701911-mv5pn 0/1 CreateContainerError 0 21m report-status-to-provider-27701912-wskz6 0/1 CreateContainerError 0 20m report-status-to-provider-27701913-lcpfm 0/1 CreateContainerError 0 19m report-status-to-provider-27701914-7lc9c 0/1 CreateContainerError 0 18m report-status-to-provider-27701915-nbr6b 0/1 CreateContainerError 0 17m report-status-to-provider-27701916-hxwwm 0/1 CreateContainerError 0 16m report-status-to-provider-27701917-46wvw 0/1 CreateContainerError 0 15m report-status-to-provider-27701918-j9x4t 0/1 CreateContainerError 0 14m report-status-to-provider-27701919-cvwj9 0/1 CreateContainerError 0 13m report-status-to-provider-27701920-9c8ns 0/1 CreateContainerError 0 12m report-status-to-provider-27701921-t69zz 0/1 CreateContainerError 0 11m report-status-to-provider-27701922-j68d7 0/1 CreateContainerError 0 10m report-status-to-provider-27701923-5dvkj 0/1 CreateContainerError 0 9m8s report-status-to-provider-27701924-pbdrz 0/1 CreateContainerError 0 8m8s report-status-to-provider-27701925-x8tlj 0/1 CreateContainerError 0 7m8s report-status-to-provider-27701926-x89gk 0/1 CreateContainerError 0 6m8s report-status-to-provider-27701927-zt7cp 0/1 CreateContainerError 0 5m8s report-status-to-provider-27701928-k9z4q 0/1 CreateContainerError 0 4m8s report-status-to-provider-27701929-vhgx4 0/1 CreateContainerError 0 3m8s report-status-to-provider-27701930-bvd86 0/1 CreateContainerError 0 2m8s report-status-to-provider-27701931-tn4lj 0/1 CreateContainerError 0 68s report-status-to-provider-27701932-4ssgd 0/1 CreateContainerError 0 8s Expected results: report-status-to-provider pod should not have any issue. Additional info: