Bug 2221995

Summary: noobaa-operator is in ErrImagePull
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Vijay Avuthu <vavuthu>
Component: odf-operatorAssignee: Boris Ranto <branto>
Status: CLOSED ERRATA QA Contact: Vijay Avuthu <vavuthu>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.14CC: branto, muagarwa, odf-bz-bot
Target Milestone: ---Keywords: Automation, Regression
Target Release: ODF 4.14.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-08 18:52:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vijay Avuthu 2023-07-11 12:33:46 UTC
Description of problem (please be detailed as possible and provide log
snippests):

not able to install mcg-operator.v4.14.0-60.stable

Version of all relevant components (if applicable):
ocs-registry:4.14.0-60

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
4/4

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. install odf 4.14 using ocs-ci
2. check csv status
3.


Actual results:

$ oc get csv
NAME                                        DISPLAY                       VERSION            REPLACES   PHASE
mcg-operator.v4.14.0-60.stable              NooBaa Operator               4.14.0-60.stable              Failed
ocs-operator.v4.14.0-60.stable              OpenShift Container Storage   4.14.0-60.stable              Succeeded
odf-csi-addons-operator.v4.14.0-60.stable   CSI Addons                    4.14.0-60.stable              Succeeded
odf-operator.v4.14.0-60.stable              OpenShift Data Foundation     4.14.0-60.stable              Succeeded



Expected results:

all csv's should in Succeeded state

Additional info:

$ oc get pods
NAME                                               READY   STATUS         RESTARTS   AGE
csi-addons-controller-manager-795fddf5fd-k6kqz     2/2     Running        0          47m
noobaa-operator-5984cb57b6-rxsxs                   1/2     ErrImagePull   0          103m
ocs-metrics-exporter-ff66c574d-7q8nz               1/1     Running        0          103m
ocs-operator-597c88f99b-nfrtp                      1/1     Running        0          103m
odf-console-58dd5c465-bfpd7                        1/1     Running        0          103m
odf-operator-controller-manager-7798f9fb5d-jgl5f   2/2     Running        0          103m
rook-ceph-operator-7bc5bdd54d-kn8t6                1/1     Running        0          103m

> noobaa-operator pod status

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    message: 'containers with unready status: [objectstorage-provisioner-sidecar]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    message: 'containers with unready status: [objectstorage-provisioner-sidecar]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://710042bfcfc84d85d76e307b94d4ac149e8e430fadde34e5909ab733794c72fb
    image: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75
    imageID: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75
    lastState: {}
    name: noobaa-operator
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-07-11T10:42:29Z"
  - image: registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342
    imageID: ""
    lastState: {}
    name: objectstorage-provisioner-sidecar
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: Back-off pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"
        reason: ImagePullBackOff
  hostIP: 10.1.112.124
  phase: Pending
  podIP: 10.128.2.17
  podIPs:
  - ip: 10.128.2.17
  qosClass: Guaranteed
  startTime: "2023-07-11T10:42:24Z"

must gather: https://url.corp.redhat.com/6ffe1b8

job: https://url.corp.redhat.com/6e8daaa

cluster is alive if live debugging is needed

Comment 3 Vijay Avuthu 2023-07-11 12:37:03 UTC
$ oc describe pod noobaa-operator-5984cb57b6-rxsxs

Events:
  Type     Reason          Age                     From               Message
  ----     ------          ----                    ----               -------
  Normal   Scheduled       113m                    default-scheduler  Successfully assigned openshift-storage/noobaa-operator-5984cb57b6-rxsxs to compute-0
  Normal   AddedInterface  113m                    multus             Add eth0 [10.128.2.17/23] from ovn-kubernetes
  Normal   Pulling         113m                    kubelet            Pulling image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75"
  Normal   Pulled          113m                    kubelet            Successfully pulled image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75" in 4.505055617s (4.505069431s including waiting)
  Normal   Created         113m                    kubelet            Created container noobaa-operator
  Normal   Started         113m                    kubelet            Started container noobaa-operator
  Warning  Failed          112m (x4 over 113m)     kubelet            Error: ImagePullBackOff
  Normal   Pulling         112m (x4 over 113m)     kubelet            Pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"
  Warning  Failed          112m (x4 over 113m)     kubelet            Failed to pull image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342": rpc error: code = Unknown desc = (Mirrors also failed: [quay.io/rhceph-dev/odf4-odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342: reading manifest sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 in quay.io/rhceph-dev/odf4-odf-cosi-sidecar-rhel9: unauthorized: access to the requested resource is not authorized]): registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342: reading manifest sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 in registry.redhat.io/odf4/odf-cosi-sidecar-rhel9: unauthorized: access to the requested resource is not authorized
  Warning  Failed          112m (x4 over 113m)     kubelet            Error: ErrImagePull
  Normal   BackOff         3m45s (x482 over 113m)  kubelet            Back-off pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"

Comment 7 Vijay Avuthu 2023-07-12 05:22:07 UTC
verified with same build ocs-registry:4.14.0-60 after fixing permissions in comment #4

job link:  https://url.corp.redhat.com/284e064

accepatnce suite passed successfully ( BUILD ID: 4.14.0-60 RUN ID: 1689093773 )

Summary

26 tests ran in 4399.99 seconds.
26 passed, 5 skipped, 0 failed, 0 errors, 0 expected failures, 0 unexpected passes

Comment 13 errata-xmlrpc 2023-11-08 18:52:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6832