Description of problem (please be detailed as possible and provide log snippests): Unable to deploy ODF4.14 on IBM Power(ppc64le) because of missing multi-arch image of objectstorage-provisioner-sidecar container in noobaa operator pod. Version of all relevant components (if applicable): ODF: 4.14.0-36.stable OCP: 4.14.0-0.nightly-ppc64le-2023-06-28-001736 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Patch default redhat-operator catalogsource. 2. Create new redhat-operators catalogsource pointing to the image: quay.io/rhceph-dev/ocs-registry:4.14.0-36 3. Login to OCP web console. Navigate to Operators -> Operatorhub 4. Search for Openshift Data foundation and install it. Actual results: ODF is not getting installed because noobaa-operator pod is failing with CrashLoopBackOff error. Expected results: ODF4.14 should be installed. Additional info:
[root@rdr-vault414-lon06-bastion-0 ~]# oc get csv -A NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-local-storage local-storage-operator.v4.14.0-202306270854 Local Storage 4.14.0-202306270854 Succeeded openshift-operator-lifecycle-manager packageserver Package Server 0.19.0 Succeeded openshift-storage mcg-operator.v4.14.0-36.stable NooBaa Operator 4.14.0-36.stable Installing openshift-storage ocs-operator.v4.14.0-36.stable OpenShift Container Storage 4.14.0-36.stable Succeeded openshift-storage odf-csi-addons-operator.v4.14.0-36.stable CSI Addons 4.14.0-36.stable Succeeded openshift-storage odf-operator.v4.14.0-36.stable OpenShift Data Foundation 4.14.0-36.stable Succeeded [root@rdr-vault414-lon06-bastion-0 ~]# oc get pods -n openshift-storage NAME READY STATUS RESTARTS AGE csi-addons-controller-manager-6b5b8449f9-gqxvm 2/2 Running 0 2m34s noobaa-operator-bf5f597c5-89kzh 1/2 CrashLoopBackOff 4 (76s ago) 2m49s ocs-metrics-exporter-57f48d7867-xv9zn 1/1 Running 0 2m35s ocs-operator-865d8cb68d-7csfg 1/1 Running 0 2m35s odf-console-7757bbbc5f-xl8kx 1/1 Running 0 2m53s odf-operator-controller-manager-5798d84c44-pm8nk 2/2 Running 0 2m53s rook-ceph-operator-5c555584cd-ncs2m 1/1 Running 0 2m35s [root@rdr-vault414-lon06-bastion-0 ~]# oc describe pod noobaa-operator-bf5f597c5-89kzh -n openshift-storage Name: noobaa-operator-bf5f597c5-89kzh Namespace: openshift-storage Priority: 0 Service Account: noobaa Node: lon06-worker-0.rdr-vault414.ibm.com/192.168.0.87 Start Time: Mon, 03 Jul 2023 12:49:27 -0400 Labels: app=noobaa noobaa-operator=deployment pod-template-hash=bf5f597c5 Annotations: alm-examples: [{"kind":"NooBaa","apiVersion":"noobaa.io/v1alpha1","metadata":{"name":"noobaa","creationTimestamp":null},"spec":{"cleanupPolicy":{},"secu... capabilities: Basic Install categories: Storage,Big Data certified: false containerImage: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726 createdAt: 2019-07-08T13:10:20.940Z description: NooBaa is an object data service for hybrid and multi cloud environments. k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.2.111/23"],"mac_address":"0a:58:0a:81:02:6f","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.111/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.2.111" ], "mac": "0a:58:0a:81:02:6f", "default": true, "dns": {} }] olm.operatorGroup: openshift-storage-5hqns olm.operatorNamespace: openshift-storage olm.skipRange: >=4.2.0 <4.14.0-36.stable olm.targetNamespaces: openshift-storage olmcahash: 67f52c7ece60d1b142a1ebbe78930d35f9601b77754fc65f1451473c3bd36cf9 openshift.io/scc: restricted-v2 operatorframework.io/properties: {"properties":[{"type":"olm.gvk","value":{"group":"noobaa.io","kind":"BackingStore","version":"v1alpha1"}},{"type":"olm.gvk","value":{"gro... operators.openshift.io/infrastructure-features: ֿ'["disconnected"]' operators.operatorframework.io/operator-type: non-standalone repository: https://github.com/noobaa/noobaa-operator seccomp.security.alpha.kubernetes.io/pod: runtime/default support: Red Hat Status: Running IP: 10.129.2.111 IPs: IP: 10.129.2.111 Controlled By: ReplicaSet/noobaa-operator-bf5f597c5 Containers: noobaa-operator: Container ID: cri-o://ea6d8dbf6fa83b99a7a6566df1a68ded4516f35c7e62ff1dd1f6858da32b60a9 Image: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726 Image ID: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726 Port: <none> Host Port: <none> State: Running Started: Mon, 03 Jul 2023 12:49:35 -0400 Ready: True Restart Count: 0 Limits: cpu: 250m memory: 512Mi Requests: cpu: 250m memory: 512Mi Environment: OPERATOR_NAME: noobaa-operator POD_NAME: noobaa-operator-bf5f597c5-89kzh (v1:metadata.name) WATCH_NAMESPACE: openshift-storage (v1:metadata.namespace) NOOBAA_CORE_IMAGE: registry.redhat.io/odf4/mcg-core-rhel9@sha256:791e360f8c0febad70bbde5797908baacb4f1a029bd90fd74734f438fe42b5fa NOOBAA_DB_IMAGE: registry.redhat.io/rhel8/postgresql-12@sha256:f7f678d44d5934ed3d95c83b4428fee4b616f37e8eadc5049778f133b4ce3713 ENABLE_NOOBAA_ADMISSION: true OPERATOR_CONDITION_NAME: mcg-operator.v4.14.0-36.stable Mounts: /apiserver.local.config/certificates from apiservice-cert (rw) /tmp/k8s-webhook-server/serving-certs from webhook-cert (rw) /var/lib/cosi from socket (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ft559 (ro) /var/run/secrets/openshift/serviceaccount from oidc-token (rw) objectstorage-provisioner-sidecar: Container ID: cri-o://5bdbf472111c00c3e5573a3da866f58d73ecff4cc3ebefc67be615ca6c298fea Image: gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387 Image ID: gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar@sha256:589c0ad4ef5d0855fe487440e634d01315bc3d883f91c44cb72577ea6e12c890 Port: <none> Host Port: <none> Args: --v=5 State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 03 Jul 2023 12:52:21 -0400 Finished: Mon, 03 Jul 2023 12:52:21 -0400 Ready: False Restart Count: 5 Limits: cpu: 100m memory: 512Mi Requests: cpu: 100m memory: 512Mi Environment: POD_NAMESPACE: openshift-storage (v1:metadata.namespace) OPERATOR_CONDITION_NAME: mcg-operator.v4.14.0-36.stable Mounts: /apiserver.local.config/certificates from apiservice-cert (rw) /tmp/k8s-webhook-server/serving-certs from webhook-cert (rw) /var/lib/cosi from socket (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ft559 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: oidc-token: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3600 socket: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> apiservice-cert: Type: Secret (a volume populated by a Secret) SecretName: noobaa-operator-service-cert Optional: false webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: noobaa-operator-service-cert Optional: false kube-api-access-ft559: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Guaranteed Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s node.ocs.openshift.io/storage=true:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m17s default-scheduler Successfully assigned openshift-storage/noobaa-operator-bf5f597c5-89kzh to lon06-worker-0.rdr-vault414.ibm.com Normal AddedInterface 3m15s multus Add eth0 [10.129.2.111/23] from ovn-kubernetes Normal Pulling 3m15s kubelet Pulling image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726" Normal Pulled 3m10s kubelet Successfully pulled image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726" in 5.412369001s (5.412384842s including waiting) Normal Created 3m9s kubelet Created container noobaa-operator Normal Started 3m9s kubelet Started container noobaa-operator Normal Pulled 3m6s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 3.240929206s (3.240946823s including waiting) Normal Pulled 3m5s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 409.291437ms (409.307338ms including waiting) Normal Pulled 2m52s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 450.403817ms (450.424934ms including waiting) Normal Pulling 2m27s (x4 over 3m9s) kubelet Pulling image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" Normal Pulled 2m27s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 431.566038ms (431.598517ms including waiting) Normal Started 2m26s (x4 over 3m6s) kubelet Started container objectstorage-provisioner-sidecar Warning BackOff 2m26s (x5 over 3m4s) kubelet Back-off restarting failed container objectstorage-provisioner-sidecar in pod noobaa-operator-bf5f597c5-89kzh_openshift-storage(c956cca7-8045-4bea-89aa-73a21665c3b9) Normal Created 2m26s (x4 over 3m6s) kubelet Created container objectstorage-provisioner-sidecar [root@rdr-vault414-lon06-bastion-0 ~]# oc logs pod/noobaa-operator-bf5f597c5-89kzh -n openshift-storage -c objectstorage-provisioner-sidecar exec /objectstorage-sidecar: exec format error [root@rdr-vault414-lon06-bastion-0 ~]# podman pull gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387 Trying to pull gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387... Getting image source signatures Copying blob 2ee0cbba6586 done Copying blob 8fdb1fc20e24 done Copying config 9e0fc3087a done Writing manifest to image destination Storing signatures WARNING: image platform (linux/amd64) does not match the expected platform (linux/ppc64le) 9e0fc3087a24a368aa71714243be37b89a74beabdb31f8d0a5c7a655e8702bbb [root@rdr-vault414-lon06-bastion-0 ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar v20221117-v0.1.0-22-g0e67387 9e0fc3087a24 7 months ago 54.2 MB [root@rdr-vault414-lon06-bastion-0 ~]# podman inspect 9e0fc3087a24 |grep Arch "Architecture": "amd64",
Talking to Deepshikha Khandelwal, she has confirmed that due to the build outage we were not able to produce a proper d/s build with the new addition (the COSI controller) and suspects this is related. We expect this to be cleared once the outage is done a new d/s build can be made
We are overriding this image now so it should be available for all the supported architectures.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6832