Bug 2221995 - noobaa-operator is in ErrImagePull
Summary: noobaa-operator is in ErrImagePull
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: odf-operator
Version: 4.14
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ODF 4.14.0
Assignee: Boris Ranto
QA Contact: Vijay Avuthu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-11 12:33 UTC by Vijay Avuthu
Modified: 2023-11-08 18:54 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-08 18:52:23 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2023:6832 0 None None None 2023-11-08 18:54:09 UTC

Description Vijay Avuthu 2023-07-11 12:33:46 UTC
Description of problem (please be detailed as possible and provide log
snippests):

not able to install mcg-operator.v4.14.0-60.stable

Version of all relevant components (if applicable):
ocs-registry:4.14.0-60

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
4/4

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Not tried

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. install odf 4.14 using ocs-ci
2. check csv status
3.


Actual results:

$ oc get csv
NAME                                        DISPLAY                       VERSION            REPLACES   PHASE
mcg-operator.v4.14.0-60.stable              NooBaa Operator               4.14.0-60.stable              Failed
ocs-operator.v4.14.0-60.stable              OpenShift Container Storage   4.14.0-60.stable              Succeeded
odf-csi-addons-operator.v4.14.0-60.stable   CSI Addons                    4.14.0-60.stable              Succeeded
odf-operator.v4.14.0-60.stable              OpenShift Data Foundation     4.14.0-60.stable              Succeeded



Expected results:

all csv's should in Succeeded state

Additional info:

$ oc get pods
NAME                                               READY   STATUS         RESTARTS   AGE
csi-addons-controller-manager-795fddf5fd-k6kqz     2/2     Running        0          47m
noobaa-operator-5984cb57b6-rxsxs                   1/2     ErrImagePull   0          103m
ocs-metrics-exporter-ff66c574d-7q8nz               1/1     Running        0          103m
ocs-operator-597c88f99b-nfrtp                      1/1     Running        0          103m
odf-console-58dd5c465-bfpd7                        1/1     Running        0          103m
odf-operator-controller-manager-7798f9fb5d-jgl5f   2/2     Running        0          103m
rook-ceph-operator-7bc5bdd54d-kn8t6                1/1     Running        0          103m

> noobaa-operator pod status

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    message: 'containers with unready status: [objectstorage-provisioner-sidecar]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    message: 'containers with unready status: [objectstorage-provisioner-sidecar]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-07-11T10:42:24Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://710042bfcfc84d85d76e307b94d4ac149e8e430fadde34e5909ab733794c72fb
    image: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75
    imageID: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75
    lastState: {}
    name: noobaa-operator
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-07-11T10:42:29Z"
  - image: registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342
    imageID: ""
    lastState: {}
    name: objectstorage-provisioner-sidecar
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: Back-off pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"
        reason: ImagePullBackOff
  hostIP: 10.1.112.124
  phase: Pending
  podIP: 10.128.2.17
  podIPs:
  - ip: 10.128.2.17
  qosClass: Guaranteed
  startTime: "2023-07-11T10:42:24Z"

must gather: https://url.corp.redhat.com/6ffe1b8

job: https://url.corp.redhat.com/6e8daaa

cluster is alive if live debugging is needed

Comment 3 Vijay Avuthu 2023-07-11 12:37:03 UTC
$ oc describe pod noobaa-operator-5984cb57b6-rxsxs

Events:
  Type     Reason          Age                     From               Message
  ----     ------          ----                    ----               -------
  Normal   Scheduled       113m                    default-scheduler  Successfully assigned openshift-storage/noobaa-operator-5984cb57b6-rxsxs to compute-0
  Normal   AddedInterface  113m                    multus             Add eth0 [10.128.2.17/23] from ovn-kubernetes
  Normal   Pulling         113m                    kubelet            Pulling image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75"
  Normal   Pulled          113m                    kubelet            Successfully pulled image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75" in 4.505055617s (4.505069431s including waiting)
  Normal   Created         113m                    kubelet            Created container noobaa-operator
  Normal   Started         113m                    kubelet            Started container noobaa-operator
  Warning  Failed          112m (x4 over 113m)     kubelet            Error: ImagePullBackOff
  Normal   Pulling         112m (x4 over 113m)     kubelet            Pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"
  Warning  Failed          112m (x4 over 113m)     kubelet            Failed to pull image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342": rpc error: code = Unknown desc = (Mirrors also failed: [quay.io/rhceph-dev/odf4-odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342: reading manifest sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 in quay.io/rhceph-dev/odf4-odf-cosi-sidecar-rhel9: unauthorized: access to the requested resource is not authorized]): registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342: reading manifest sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 in registry.redhat.io/odf4/odf-cosi-sidecar-rhel9: unauthorized: access to the requested resource is not authorized
  Warning  Failed          112m (x4 over 113m)     kubelet            Error: ErrImagePull
  Normal   BackOff         3m45s (x482 over 113m)  kubelet            Back-off pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"

Comment 7 Vijay Avuthu 2023-07-12 05:22:07 UTC
verified with same build ocs-registry:4.14.0-60 after fixing permissions in comment #4

job link:  https://url.corp.redhat.com/284e064

accepatnce suite passed successfully ( BUILD ID: 4.14.0-60 RUN ID: 1689093773 )

Summary

26 tests ran in 4399.99 seconds.
26 passed, 5 skipped, 0 failed, 0 errors, 0 expected failures, 0 unexpected passes

Comment 13 errata-xmlrpc 2023-11-08 18:52:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6832


Note You need to log in before you can comment on or make changes to this bug.