Description of problem (please be detailed as possible and provide log snippests): upgrade from ODF 4.10.13 to ODF 4.11.9 is failing on IBM Power with Imagepullbackoff error. Version of all relevant components (if applicable): [root@nara-odf411-syd05-bastion-0 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.43 True False 8h Cluster version is 4.11.43 [root@nara-odf411-syd05-bastion-0 ~]# ODF version : 4.11.9-2 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes Is there any workaround available to the best of your knowledge? NO Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. create OCP 4.10 cluster and deploy ODF 4.10.13 2. upgrade OCP to 4.11 3. create catalog source for ODF 4.11.9 and upgrade the subscription to 4.11.9 Actual results: ODF upgrade is failing with two pods in imagepullback error Expected results: ODF upgrade should succeed Additional info:
[root@nara2-odf411-syd05-bastion-0 ~]# oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.10.13 NooBaa Operator 4.10.13 mcg-operator.v4.10.12 Succeeded ocs-operator.v4.10.13 OpenShift Container Storage 4.10.13 ocs-operator.v4.10.12 Succeeded odf-csi-addons-operator.v4.10.13 CSI Addons 4.10.13 odf-csi-addons-operator.v4.10.12 Succeeded odf-operator.v4.10.13 OpenShift Data Foundation 4.10.13 odf-operator.v4.10.12 Replacing odf-operator.v4.11.9 OpenShift Data Foundation 4.11.9 odf-operator.v4.10.13 Failed [root@nara2-odf411-syd05-bastion-0 ~]# #oc get pods odf-console-5c5d759cb5-dkmvr 0/1 ImagePullBackOff 0 14m odf-operator-controller-manager-5f6bf74796-fbbb6 1/2 ImagePullBackOff 0 14m root@nara2-odf411-syd05-bastion-0 ~]# oc describe pod odf-console-5c5d759cb5-dkmvr Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned openshift-storage/odf-console-5c5d759cb5-dkmvr to syd05-worker-2.nara2-odf411.ibm.com by syd05-master-0.nara2-odf411.ibm.com Normal AddedInterface 15m multus Add eth0 [10.131.0.164/23] from openshift-sdn Normal Pulling 13m (x4 over 15m) kubelet Pulling image "registry.redhat.io/odf4/odf-console-rhel8@sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4" Warning Failed 13m (x4 over 15m) kubelet Failed to pull image "registry.redhat.io/odf4/odf-console-rhel8@sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4": rpc error: code = Unknown desc = reading manifest sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4 in registry.redhat.io/odf4/odf-console-rhel8: manifest unknown: manifest unknown Warning Failed 13m (x4 over 15m) kubelet Error: ErrImagePull Warning Failed 13m (x6 over 15m) kubelet Error: ImagePullBackOff Normal BackOff 2s (x65 over 15m) kubelet Back-off pulling image "registry.redhat.io/odf4/odf-console-rhel8@sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4" [root@nara2-odf411-syd05-bastion-0 ~]# oc describe pod odf-operator-controller-manager-5f6bf74796-fbbb6 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned openshift-storage/odf-operator-controller-manager-5f6bf74796-fbbb6 to syd05-worker-2.nara2-odf411.ibm.com by syd05-master-0.nara2-odf411.ibm.com Normal AddedInterface 15m multus Add eth0 [10.131.0.163/23] from openshift-sdn Normal Pulling 15m kubelet Pulling image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:247a74298ff310d6e1992e21041e034e46fabaa5230d8ed553db27352d4441b5" Normal Pulled 15m kubelet Successfully pulled image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:247a74298ff310d6e1992e21041e034e46fabaa5230d8ed553db27352d4441b5" in 4.408432605s Normal Created 15m kubelet Created container kube-rbac-proxy Normal Started 15m kubelet Started container kube-rbac-proxy Warning Failed 14m (x3 over 15m) kubelet Error: ErrImagePull Warning Failed 14m (x5 over 15m) kubelet Error: ImagePullBackOff Normal Pulling 14m (x4 over 15m) kubelet Pulling image "registry.redhat.io/odf4/odf-rhel8-operator@sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8" Warning Failed 14m (x4 over 15m) kubelet Failed to pull image "registry.redhat.io/odf4/odf-rhel8-operator@sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8": rpc error: code = Unknown desc = reading manifest sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8 in registry.redhat.io/odf4/odf-rhel8-operator: manifest unknown: manifest unknown Normal BackOff 33s (x63 over 15m) kubelet Back-off pulling image "registry.redhat.io/odf4/odf-rhel8-operator@sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8" [root@nara2-odf411-syd05-bastion-0 ~]#
Hi Kunal we are using the quay.io in our catalog source for ODF operator. we tried both manual upgrade and also through the script both are giving same results. 1. create 4.10 version OCP cluster. 2. deploy ODF 4.10.13 3. upgrade OCP to 4.11 4. update catalog source for 4.11.9 for ODF operator 5. update channel in subscription for ODF operator to 4.11 6. The upgrade is failed due to odf-console and odf-operator-controller-manager pods are failing. the cluster got deleted over weekend. if required we can setup. mustgather logs will be shared soon.
mustgather logs are placed at below location: https://drive.google.com/file/d/1uM1lZajtfn6scSj3I_5EoQwkbAi7QswP/view?usp=sharing
after updating the ImageContentSourcePolicy , the image pull issue is resolved and upgrade is working fine. Thanks for the help. we can close this case.
found this as well. https://github.com/red-hat-storage/ocs-ci/pull/7836/files