Description of problem (please be detailed as possible and provide log snippests): not able to install mcg-operator.v4.14.0-60.stable Version of all relevant components (if applicable): ocs-registry:4.14.0-60 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes Is there any workaround available to the best of your knowledge? NA Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 4/4 Can this issue reproducible? Yes Can this issue reproduce from the UI? Not tried If this is a regression, please provide more details to justify this: Yes Steps to Reproduce: 1. install odf 4.14 using ocs-ci 2. check csv status 3. Actual results: $ oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.14.0-60.stable NooBaa Operator 4.14.0-60.stable Failed ocs-operator.v4.14.0-60.stable OpenShift Container Storage 4.14.0-60.stable Succeeded odf-csi-addons-operator.v4.14.0-60.stable CSI Addons 4.14.0-60.stable Succeeded odf-operator.v4.14.0-60.stable OpenShift Data Foundation 4.14.0-60.stable Succeeded Expected results: all csv's should in Succeeded state Additional info: $ oc get pods NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-795fddf5fd-k6kqz 2/2 Running 0 47m noobaa-operator-5984cb57b6-rxsxs 1/2 ErrImagePull 0 103m ocs-metrics-exporter-ff66c574d-7q8nz 1/1 Running 0 103m ocs-operator-597c88f99b-nfrtp 1/1 Running 0 103m odf-console-58dd5c465-bfpd7 1/1 Running 0 103m odf-operator-controller-manager-7798f9fb5d-jgl5f 2/2 Running 0 103m rook-ceph-operator-7bc5bdd54d-kn8t6 1/1 Running 0 103m > noobaa-operator pod status status: conditions: - lastProbeTime: null lastTransitionTime: "2023-07-11T10:42:24Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-07-11T10:42:24Z" message: 'containers with unready status: [objectstorage-provisioner-sidecar]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-07-11T10:42:24Z" message: 'containers with unready status: [objectstorage-provisioner-sidecar]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-07-11T10:42:24Z" status: "True" type: PodScheduled containerStatuses: - containerID: cri-o://710042bfcfc84d85d76e307b94d4ac149e8e430fadde34e5909ab733794c72fb image: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75 imageID: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75 lastState: {} name: noobaa-operator ready: true restartCount: 0 started: true state: running: startedAt: "2023-07-11T10:42:29Z" - image: registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 imageID: "" lastState: {} name: objectstorage-provisioner-sidecar ready: false restartCount: 0 started: false state: waiting: message: Back-off pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342" reason: ImagePullBackOff hostIP: 10.1.112.124 phase: Pending podIP: 10.128.2.17 podIPs: - ip: 10.128.2.17 qosClass: Guaranteed startTime: "2023-07-11T10:42:24Z" must gather: https://url.corp.redhat.com/6ffe1b8 job: https://url.corp.redhat.com/6e8daaa cluster is alive if live debugging is needed
$ oc describe pod noobaa-operator-5984cb57b6-rxsxs Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 113m default-scheduler Successfully assigned openshift-storage/noobaa-operator-5984cb57b6-rxsxs to compute-0 Normal AddedInterface 113m multus Add eth0 [10.128.2.17/23] from ovn-kubernetes Normal Pulling 113m kubelet Pulling image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75" Normal Pulled 113m kubelet Successfully pulled image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:53a9c21bcee200809ef2b25aef2ed6be07605a00e6107d37c43878542b2f8d75" in 4.505055617s (4.505069431s including waiting) Normal Created 113m kubelet Created container noobaa-operator Normal Started 113m kubelet Started container noobaa-operator Warning Failed 112m (x4 over 113m) kubelet Error: ImagePullBackOff Normal Pulling 112m (x4 over 113m) kubelet Pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342" Warning Failed 112m (x4 over 113m) kubelet Failed to pull image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342": rpc error: code = Unknown desc = (Mirrors also failed: [quay.io/rhceph-dev/odf4-odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342: reading manifest sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 in quay.io/rhceph-dev/odf4-odf-cosi-sidecar-rhel9: unauthorized: access to the requested resource is not authorized]): registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342: reading manifest sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342 in registry.redhat.io/odf4/odf-cosi-sidecar-rhel9: unauthorized: access to the requested resource is not authorized Warning Failed 112m (x4 over 113m) kubelet Error: ErrImagePull Normal BackOff 3m45s (x482 over 113m) kubelet Back-off pulling image "registry.redhat.io/odf4/odf-cosi-sidecar-rhel9@sha256:1181eed3b90b289d67a718b678769425a5e7c73daf853a22a4818ab5c1768342"
verified with same build ocs-registry:4.14.0-60 after fixing permissions in comment #4 job link: https://url.corp.redhat.com/284e064 accepatnce suite passed successfully ( BUILD ID: 4.14.0-60 RUN ID: 1689093773 ) Summary 26 tests ran in 4399.99 seconds. 26 passed, 5 skipped, 0 failed, 0 errors, 0 expected failures, 0 unexpected passes
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6832