Bug 2222955
| Summary: | upgrade from ODF 4.10.13 to ODF 4.11.9 is failing on IBM Power | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | narayanspg <ngowda> |
| Component: | odf-operator | Assignee: | Nitin Goyal <nigoyal> |
| Status: | CLOSED NOTABUG | QA Contact: | Elad <ebenahar> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.11 | CC: | bkunal, muagarwa, ngowda, ocs-bugs, odf-bz-bot |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | ppc64le | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-07-17 09:37:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
narayanspg
2023-07-14 15:18:41 UTC
[root@nara2-odf411-syd05-bastion-0 ~]# oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.10.13 NooBaa Operator 4.10.13 mcg-operator.v4.10.12 Succeeded ocs-operator.v4.10.13 OpenShift Container Storage 4.10.13 ocs-operator.v4.10.12 Succeeded odf-csi-addons-operator.v4.10.13 CSI Addons 4.10.13 odf-csi-addons-operator.v4.10.12 Succeeded odf-operator.v4.10.13 OpenShift Data Foundation 4.10.13 odf-operator.v4.10.12 Replacing odf-operator.v4.11.9 OpenShift Data Foundation 4.11.9 odf-operator.v4.10.13 Failed [root@nara2-odf411-syd05-bastion-0 ~]# #oc get pods odf-console-5c5d759cb5-dkmvr 0/1 ImagePullBackOff 0 14m odf-operator-controller-manager-5f6bf74796-fbbb6 1/2 ImagePullBackOff 0 14m root@nara2-odf411-syd05-bastion-0 ~]# oc describe pod odf-console-5c5d759cb5-dkmvr Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned openshift-storage/odf-console-5c5d759cb5-dkmvr to syd05-worker-2.nara2-odf411.ibm.com by syd05-master-0.nara2-odf411.ibm.com Normal AddedInterface 15m multus Add eth0 [10.131.0.164/23] from openshift-sdn Normal Pulling 13m (x4 over 15m) kubelet Pulling image "registry.redhat.io/odf4/odf-console-rhel8@sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4" Warning Failed 13m (x4 over 15m) kubelet Failed to pull image "registry.redhat.io/odf4/odf-console-rhel8@sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4": rpc error: code = Unknown desc = reading manifest sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4 in registry.redhat.io/odf4/odf-console-rhel8: manifest unknown: manifest unknown Warning Failed 13m (x4 over 15m) kubelet Error: ErrImagePull Warning Failed 13m (x6 over 15m) kubelet Error: ImagePullBackOff Normal BackOff 2s (x65 over 15m) kubelet Back-off pulling image "registry.redhat.io/odf4/odf-console-rhel8@sha256:59bec5fa0cd4102b0d6d7b75919f4a0376b7ab397ea82c4389685a23e37c37d4" [root@nara2-odf411-syd05-bastion-0 ~]# oc describe pod odf-operator-controller-manager-5f6bf74796-fbbb6 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned openshift-storage/odf-operator-controller-manager-5f6bf74796-fbbb6 to syd05-worker-2.nara2-odf411.ibm.com by syd05-master-0.nara2-odf411.ibm.com Normal AddedInterface 15m multus Add eth0 [10.131.0.163/23] from openshift-sdn Normal Pulling 15m kubelet Pulling image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:247a74298ff310d6e1992e21041e034e46fabaa5230d8ed553db27352d4441b5" Normal Pulled 15m kubelet Successfully pulled image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:247a74298ff310d6e1992e21041e034e46fabaa5230d8ed553db27352d4441b5" in 4.408432605s Normal Created 15m kubelet Created container kube-rbac-proxy Normal Started 15m kubelet Started container kube-rbac-proxy Warning Failed 14m (x3 over 15m) kubelet Error: ErrImagePull Warning Failed 14m (x5 over 15m) kubelet Error: ImagePullBackOff Normal Pulling 14m (x4 over 15m) kubelet Pulling image "registry.redhat.io/odf4/odf-rhel8-operator@sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8" Warning Failed 14m (x4 over 15m) kubelet Failed to pull image "registry.redhat.io/odf4/odf-rhel8-operator@sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8": rpc error: code = Unknown desc = reading manifest sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8 in registry.redhat.io/odf4/odf-rhel8-operator: manifest unknown: manifest unknown Normal BackOff 33s (x63 over 15m) kubelet Back-off pulling image "registry.redhat.io/odf4/odf-rhel8-operator@sha256:2bd829ca62a6a92b70440e067da9931977e71f2dd0ff4a71b4fc8fc708b930e8" [root@nara2-odf411-syd05-bastion-0 ~]# Hi Kunal we are using the quay.io in our catalog source for ODF operator. we tried both manual upgrade and also through the script both are giving same results. 1. create 4.10 version OCP cluster. 2. deploy ODF 4.10.13 3. upgrade OCP to 4.11 4. update catalog source for 4.11.9 for ODF operator 5. update channel in subscription for ODF operator to 4.11 6. The upgrade is failed due to odf-console and odf-operator-controller-manager pods are failing. the cluster got deleted over weekend. if required we can setup. mustgather logs will be shared soon. mustgather logs are placed at below location: https://drive.google.com/file/d/1uM1lZajtfn6scSj3I_5EoQwkbAi7QswP/view?usp=sharing after updating the ImageContentSourcePolicy , the image pull issue is resolved and upgrade is working fine. Thanks for the help. we can close this case. found this as well. https://github.com/red-hat-storage/ocs-ci/pull/7836/files |