Description of problem (please be detailed as possible and provide log snippests): odf-operator.v4.9.0-136.ci fails to install with odf-console in ImagePullBackOff Version of all relevant components (if applicable): odf-operator.v4.9.0-136 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? yes Is there any workaround available to the best of your knowledge? NA Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? 1/1 Can this issue reproduce from the UI? Not Tried If this is a regression, please provide more details to justify this: Yes Steps to Reproduce: 1. Install OCS using ocs-ci 2. verify odf operator in succeeded phase 3. Actual results: odf-operator.v4.9.0-136.ci in Failed state Expected results: odf-operator.v4.9.0-136.ci should be in Succeeded state Additional info: $ oc get csv NAME DISPLAY VERSION REPLACES PHASE noobaa-operator.v4.9.0-136.ci NooBaa Operator 4.9.0-136.ci Succeeded ocs-operator.v4.9.0-136.ci OpenShift Container Storage 4.9.0-136.ci Succeeded odf-operator.v4.9.0-136.ci OpenShift Data Foundation 4.9.0-136.ci Failed > $ oc describe csv odf-operator.v4.9.0-136.ci Name: odf-operator.v4.9.0-136.ci Namespace: openshift-storage Labels: olm.api.62e2d1ee37777c10=provided operatorframework.io/arch.amd64=supported operatorframework.io/arch.ppc64le=supported operatorframework.io/arch.s390x=supported operators.coreos.com/odf-operator.openshift-storage= Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 9m7s operator-lifecycle-manager requirements not yet checked Normal RequirementsNotMet 9m6s operator-lifecycle-manager one or more requirements couldn't be found Normal InstallWaiting 9m2s operator-lifecycle-manager installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability. Warning InstallCheckFailed 4m4s (x2 over 4m4s) operator-lifecycle-manager install timeout Normal NeedsReinstall 4m3s (x2 over 4m3s) operator-lifecycle-manager installing: waiting for deployment odf-console to become ready: deployment "odf-console" not available: Deployment does not have minimum availability. Normal AllRequirementsMet 4m2s (x4 over 9m5s) operator-lifecycle-manager all requirements found, attempting install Normal InstallSucceeded 4m2s (x3 over 9m4s) operator-lifecycle-manager waiting for install components to report healthy Normal InstallWaiting 4m2s operator-lifecycle-manager installing: waiting for deployment odf-console to become ready: waiting for spec update of deployment "odf-console" to be observed... Normal InstallWaiting 4m1s (x5 over 8m33s) operator-lifecycle-manager installing: waiting for deployment odf-console to become ready: deployment "odf-console" not available: Deployment does not have > pods $ oc get pods NAME READY STATUS RESTARTS AGE noobaa-operator-f889d95d5-lc7nz 1/1 Running 0 9m46s ocs-metrics-exporter-7d54f6587-hcx62 1/1 Running 0 9m35s ocs-operator-75d48c687b-vr7x6 1/1 Running 0 9m36s odf-console-84d97d7857-wckws 1/2 ImagePullBackOff 0 9m37s odf-operator-controller-manager-5b77594db9-svr4h 2/2 Running 0 9m37s rook-ceph-operator-558bdd4dbc-btz4p 1/1 Running 0 9m36s > $ oc describe pod odf-console-84d97d7857-wckws Name: odf-console-84d97d7857-wckws Namespace: openshift-storage Priority: 0 Node: compute-0/10.1.160.120 Start Time: Tue, 14 Sep 2021 11:48:30 +0530 Labels: app=odf-console pod-template-hash=84d97d7857 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m55s default-scheduler Successfully assigned openshift-storage/odf-console-84d97d7857-wckws to compute-0 Warning FailedMount 9m54s kubelet MountVolume.SetUp failed for volume "odf-console-serving-cert" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 9m54s kubelet MountVolume.SetUp failed for volume "ibm-console-serving-cert" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 9m39s (x5 over 9m54s) kubelet MountVolume.SetUp failed for volume "ibm-console-serving-cert" : secret "ibm-console-serving-cert" not found Warning FailedMount 9m39s (x5 over 9m54s) kubelet MountVolume.SetUp failed for volume "odf-console-serving-cert" : secret "odf-console-serving-cert" not found Normal AddedInterface 9m21s multus Add eth0 [10.129.2.17/23] from openshift-sdn Warning Failed 9m20s kubelet Error: ErrImagePull Normal Pulling 9m20s kubelet Pulling image "docker.io/ibmcom/ibm-storage-odf-plugin:0.2.1" Normal Created 9m16s kubelet Created container ibm-console Normal Started 9m16s kubelet Started container ibm-console Normal Pulled 9m16s kubelet Successfully pulled image "docker.io/ibmcom/ibm-storage-odf-plugin:0.2.1" in 3.837284629s Warning Failed 9m14s (x2 over 9m15s) kubelet Error: ImagePullBackOff Normal Pulling 8m59s (x2 over 9m21s) kubelet Pulling image "quay.io/ocs-dev/odf-console:4.9-13.93f7248.master" Warning Failed 8m59s (x2 over 9m20s) kubelet Failed to pull image "quay.io/ocs-dev/odf-console:4.9-13.93f7248.master": rpc error: code = Unknown desc = reading manifest 4.9-13.93f7248.master in quay.io/ocs-dev/odf-console: manifest unknown: manifest unknown Normal BackOff 4m44s (x19 over 9m15s) kubelet Back-off pulling image "quay.io/ocs-dev/odf-console:4.9-13.93f7248.master" Job: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/5926/console
must gather logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthue-odf/vavuthue-odf_20210914T051504/logs/failed_testcase_ocs_logs_1631598266/deployment_ocs_logs/
This should be fixed by: https://gitlab.cee.redhat.com/ceph/rhcs-jenkins-jobs/-/merge_requests/720
verified with odf-operator.v4.9.0-139.ci 14:48:21 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 5 seconds before next iteration 14:48:26 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get csv odf-operator.v4.9.0-139.ci -n openshift-storage -o yaml 14:48:31 - MainThread - ocs_ci.ocs.ocp - INFO - Resource odf-operator.v4.9.0-139.ci is in phase: Succeeded! job: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/5973/consoleFull
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:5086