Bug 2003904

Summary:	odf-operator.v4.9.0-136.ci fails to install with odf-console in ImagePullBackOff
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Vijay Avuthu <vavuthu>
Component:	build	Assignee:	Boris Ranto <branto>
Status:	CLOSED ERRATA	QA Contact:	Vijay Avuthu <vavuthu>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.9	CC:	bkunal, bniver, branto, jarrpa, jijoy, madam, muagarwa, ocs-bugs, odf-bz-bot, sostapov
Target Milestone:	---	Keywords:	Automation, Regression, TestBlocker
Target Release:	ODF 4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.9.0-138.ci	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-12-13 17:46:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vijay Avuthu 2021-09-14 06:35:06 UTC

Description of problem (please be detailed as possible and provide log
snippests):

odf-operator.v4.9.0-136.ci fails to install with odf-console in ImagePullBackOff

Version of all relevant components (if applicable):
odf-operator.v4.9.0-136


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes

Is there any workaround available to the best of your knowledge?
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
1/1

Can this issue reproduce from the UI?
Not Tried

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. Install OCS using ocs-ci
2. verify odf operator in succeeded phase
3.


Actual results:

odf-operator.v4.9.0-136.ci in Failed state


Expected results:

odf-operator.v4.9.0-136.ci should be in Succeeded state

Additional info:

$ oc get csv
NAME                            DISPLAY                       VERSION        REPLACES   PHASE
noobaa-operator.v4.9.0-136.ci   NooBaa Operator               4.9.0-136.ci              Succeeded
ocs-operator.v4.9.0-136.ci      OpenShift Container Storage   4.9.0-136.ci              Succeeded
odf-operator.v4.9.0-136.ci      OpenShift Data Foundation     4.9.0-136.ci              Failed


> $ oc describe csv odf-operator.v4.9.0-136.ci
Name:         odf-operator.v4.9.0-136.ci
Namespace:    openshift-storage
Labels:       olm.api.62e2d1ee37777c10=provided
              operatorframework.io/arch.amd64=supported
              operatorframework.io/arch.ppc64le=supported
              operatorframework.io/arch.s390x=supported
              operators.coreos.com/odf-operator.openshift-storage=

Events:
  Type     Reason               Age                   From                        Message
  ----     ------               ----                  ----                        -------
  Normal   RequirementsUnknown  9m7s                  operator-lifecycle-manager  requirements not yet checked
  Normal   RequirementsNotMet   9m6s                  operator-lifecycle-manager  one or more requirements couldn't be found
  Normal   InstallWaiting       9m2s                  operator-lifecycle-manager  installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability.
  Warning  InstallCheckFailed   4m4s (x2 over 4m4s)   operator-lifecycle-manager  install timeout
  Normal   NeedsReinstall       4m3s (x2 over 4m3s)   operator-lifecycle-manager  installing: waiting for deployment odf-console to become ready: deployment "odf-console" not available: Deployment does not have minimum availability.
  Normal   AllRequirementsMet   4m2s (x4 over 9m5s)   operator-lifecycle-manager  all requirements found, attempting install
  Normal   InstallSucceeded     4m2s (x3 over 9m4s)   operator-lifecycle-manager  waiting for install components to report healthy
  Normal   InstallWaiting       4m2s                  operator-lifecycle-manager  installing: waiting for deployment odf-console to become ready: waiting for spec update of deployment "odf-console" to be observed...
  Normal   InstallWaiting       4m1s (x5 over 8m33s)  operator-lifecycle-manager  installing: waiting for deployment odf-console to become ready: deployment "odf-console" not available: Deployment does not have

> pods

$ oc get pods
NAME                                               READY   STATUS             RESTARTS   AGE
noobaa-operator-f889d95d5-lc7nz                    1/1     Running            0          9m46s
ocs-metrics-exporter-7d54f6587-hcx62               1/1     Running            0          9m35s
ocs-operator-75d48c687b-vr7x6                      1/1     Running            0          9m36s
odf-console-84d97d7857-wckws                       1/2     ImagePullBackOff   0          9m37s
odf-operator-controller-manager-5b77594db9-svr4h   2/2     Running            0          9m37s
rook-ceph-operator-558bdd4dbc-btz4p                1/1     Running            0          9m36s


> $ oc describe pod odf-console-84d97d7857-wckws
Name:         odf-console-84d97d7857-wckws
Namespace:    openshift-storage
Priority:     0
Node:         compute-0/10.1.160.120
Start Time:   Tue, 14 Sep 2021 11:48:30 +0530
Labels:       app=odf-console
              pod-template-hash=84d97d7857

Events:
  Type     Reason          Age                     From               Message
  ----     ------          ----                    ----               -------
  Normal   Scheduled       9m55s                   default-scheduler  Successfully assigned openshift-storage/odf-console-84d97d7857-wckws to compute-0
  Warning  FailedMount     9m54s                   kubelet            MountVolume.SetUp failed for volume "odf-console-serving-cert" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount     9m54s                   kubelet            MountVolume.SetUp failed for volume "ibm-console-serving-cert" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount     9m39s (x5 over 9m54s)   kubelet            MountVolume.SetUp failed for volume "ibm-console-serving-cert" : secret "ibm-console-serving-cert" not found
  Warning  FailedMount     9m39s (x5 over 9m54s)   kubelet            MountVolume.SetUp failed for volume "odf-console-serving-cert" : secret "odf-console-serving-cert" not found
  Normal   AddedInterface  9m21s                   multus             Add eth0 [10.129.2.17/23] from openshift-sdn
  Warning  Failed          9m20s                   kubelet            Error: ErrImagePull
  Normal   Pulling         9m20s                   kubelet            Pulling image "docker.io/ibmcom/ibm-storage-odf-plugin:0.2.1"
  Normal   Created         9m16s                   kubelet            Created container ibm-console
  Normal   Started         9m16s                   kubelet            Started container ibm-console
  Normal   Pulled          9m16s                   kubelet            Successfully pulled image "docker.io/ibmcom/ibm-storage-odf-plugin:0.2.1" in 3.837284629s
  Warning  Failed          9m14s (x2 over 9m15s)   kubelet            Error: ImagePullBackOff
  Normal   Pulling         8m59s (x2 over 9m21s)   kubelet            Pulling image "quay.io/ocs-dev/odf-console:4.9-13.93f7248.master"
  Warning  Failed          8m59s (x2 over 9m20s)   kubelet            Failed to pull image "quay.io/ocs-dev/odf-console:4.9-13.93f7248.master": rpc error: code = Unknown desc = reading manifest 4.9-13.93f7248.master in quay.io/ocs-dev/odf-console: manifest unknown: manifest unknown
  Normal   BackOff         4m44s (x19 over 9m15s)  kubelet            Back-off pulling image "quay.io/ocs-dev/odf-console:4.9-13.93f7248.master"


Job: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/5926/console

Comment 3 Vijay Avuthu 2021-09-14 07:16:31 UTC

must gather logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthue-odf/vavuthue-odf_20210914T051504/logs/failed_testcase_ocs_logs_1631598266/deployment_ocs_logs/

Comment 5 Boris Ranto 2021-09-14 07:55:22 UTC

This should be fixed by:

https://gitlab.cee.redhat.com/ceph/rhcs-jenkins-jobs/-/merge_requests/720

Comment 8 Vijay Avuthu 2021-09-24 04:27:51 UTC

verified with odf-operator.v4.9.0-139.ci

14:48:21 - MainThread - ocs_ci.utility.utils - INFO - Going to sleep for 5 seconds before next iteration
14:48:26 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get csv odf-operator.v4.9.0-139.ci -n openshift-storage -o yaml
14:48:31 - MainThread - ocs_ci.ocs.ocp - INFO - Resource odf-operator.v4.9.0-139.ci is in phase: Succeeded!

job: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/5973/consoleFull

Comment 14 errata-xmlrpc 2021-12-13 17:46:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:5086