Bug 2219436

Summary:	Unable to deploy ODF4.14 on IBM Power(ppc64le) because of missing multiarch image of objectstorage-provisioner-sidecar in noobaa operator pod
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Aaruni Aggarwal <aaaggarw>
Component:	build	Assignee:	Tamil <tmuthami>
Status:	VERIFIED ---	QA Contact:	Petr Balogh <pbalogh>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.14	CC:	branto, ebenahar, muagarwa, nbecker, odf-bz-bot
Target Milestone:	---	Keywords:	Regression
Target Release:	---
Hardware:	ppc64le
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Aaruni Aggarwal 2023-07-03 17:05:51 UTC

Description of problem (please be detailed as possible and provide log
snippests):

Unable to deploy ODF4.14 on IBM Power(ppc64le) because of missing multi-arch image of objectstorage-provisioner-sidecar container in noobaa operator pod.

Version of all relevant components (if applicable):

ODF: 4.14.0-36.stable
OCP: 4.14.0-0.nightly-ppc64le-2023-06-28-001736

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Patch default redhat-operator catalogsource.

2. Create new redhat-operators catalogsource pointing to the image: quay.io/rhceph-dev/ocs-registry:4.14.0-36

3. Login to OCP web console. Navigate to Operators -> Operatorhub

4. Search for Openshift Data foundation and install it.


Actual results:

ODF is not getting installed because noobaa-operator pod is failing with CrashLoopBackOff error. 

Expected results:
ODF4.14 should be installed.

Additional info:

Comment 2 Aaruni Aggarwal 2023-07-03 17:09:17 UTC

[root@rdr-vault414-lon06-bastion-0 ~]# oc get csv -A
NAMESPACE                              NAME                                          DISPLAY                       VERSION               REPLACES   PHASE
openshift-local-storage                local-storage-operator.v4.14.0-202306270854   Local Storage                 4.14.0-202306270854              Succeeded
openshift-operator-lifecycle-manager   packageserver                                 Package Server                0.19.0                           Succeeded
openshift-storage                      mcg-operator.v4.14.0-36.stable                NooBaa Operator               4.14.0-36.stable                 Installing
openshift-storage                      ocs-operator.v4.14.0-36.stable                OpenShift Container Storage   4.14.0-36.stable                 Succeeded
openshift-storage                      odf-csi-addons-operator.v4.14.0-36.stable     CSI Addons                    4.14.0-36.stable                 Succeeded
openshift-storage                      odf-operator.v4.14.0-36.stable                OpenShift Data Foundation     4.14.0-36.stable                 Succeeded

[root@rdr-vault414-lon06-bastion-0 ~]# oc get pods -n openshift-storage
NAME                                               READY   STATUS             RESTARTS      AGE
csi-addons-controller-manager-6b5b8449f9-gqxvm     2/2     Running            0             2m34s
noobaa-operator-bf5f597c5-89kzh                    1/2     CrashLoopBackOff   4 (76s ago)   2m49s
ocs-metrics-exporter-57f48d7867-xv9zn              1/1     Running            0             2m35s
ocs-operator-865d8cb68d-7csfg                      1/1     Running            0             2m35s
odf-console-7757bbbc5f-xl8kx                       1/1     Running            0             2m53s
odf-operator-controller-manager-5798d84c44-pm8nk   2/2     Running            0             2m53s
rook-ceph-operator-5c555584cd-ncs2m                1/1     Running            0             2m35s

[root@rdr-vault414-lon06-bastion-0 ~]# oc describe pod noobaa-operator-bf5f597c5-89kzh -n openshift-storage
Name:             noobaa-operator-bf5f597c5-89kzh
Namespace:        openshift-storage
Priority:         0
Service Account:  noobaa
Node:             lon06-worker-0.rdr-vault414.ibm.com/192.168.0.87
Start Time:       Mon, 03 Jul 2023 12:49:27 -0400
Labels:           app=noobaa
                  noobaa-operator=deployment
                  pod-template-hash=bf5f597c5
Annotations:      alm-examples:
                    [{"kind":"NooBaa","apiVersion":"noobaa.io/v1alpha1","metadata":{"name":"noobaa","creationTimestamp":null},"spec":{"cleanupPolicy":{},"secu...
                  capabilities: Basic Install
                  categories: Storage,Big Data
                  certified: false
                  containerImage: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726
                  createdAt: 2019-07-08T13:10:20.940Z
                  description: NooBaa is an object data service for hybrid and multi cloud environments.
                  k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["10.129.2.111/23"],"mac_address":"0a:58:0a:81:02:6f","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.111/2...
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "10.129.2.111"
                        ],
                        "mac": "0a:58:0a:81:02:6f",
                        "default": true,
                        "dns": {}
                    }]
                  olm.operatorGroup: openshift-storage-5hqns
                  olm.operatorNamespace: openshift-storage
                  olm.skipRange: >=4.2.0 <4.14.0-36.stable
                  olm.targetNamespaces: openshift-storage
                  olmcahash: 67f52c7ece60d1b142a1ebbe78930d35f9601b77754fc65f1451473c3bd36cf9
                  openshift.io/scc: restricted-v2
                  operatorframework.io/properties:
                    {"properties":[{"type":"olm.gvk","value":{"group":"noobaa.io","kind":"BackingStore","version":"v1alpha1"}},{"type":"olm.gvk","value":{"gro...
                  operators.openshift.io/infrastructure-features: ֿ'["disconnected"]'
                  operators.operatorframework.io/operator-type: non-standalone
                  repository: https://github.com/noobaa/noobaa-operator
                  seccomp.security.alpha.kubernetes.io/pod: runtime/default
                  support: Red Hat
Status:           Running
IP:               10.129.2.111
IPs:
  IP:           10.129.2.111
Controlled By:  ReplicaSet/noobaa-operator-bf5f597c5
Containers:
  noobaa-operator:
    Container ID:   cri-o://ea6d8dbf6fa83b99a7a6566df1a68ded4516f35c7e62ff1dd1f6858da32b60a9
    Image:          registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726
    Image ID:       registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 03 Jul 2023 12:49:35 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     250m
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  512Mi
    Environment:
      OPERATOR_NAME:            noobaa-operator
      POD_NAME:                 noobaa-operator-bf5f597c5-89kzh (v1:metadata.name)
      WATCH_NAMESPACE:          openshift-storage (v1:metadata.namespace)
      NOOBAA_CORE_IMAGE:        registry.redhat.io/odf4/mcg-core-rhel9@sha256:791e360f8c0febad70bbde5797908baacb4f1a029bd90fd74734f438fe42b5fa
      NOOBAA_DB_IMAGE:          registry.redhat.io/rhel8/postgresql-12@sha256:f7f678d44d5934ed3d95c83b4428fee4b616f37e8eadc5049778f133b4ce3713
      ENABLE_NOOBAA_ADMISSION:  true
      OPERATOR_CONDITION_NAME:  mcg-operator.v4.14.0-36.stable
    Mounts:
      /apiserver.local.config/certificates from apiservice-cert (rw)
      /tmp/k8s-webhook-server/serving-certs from webhook-cert (rw)
      /var/lib/cosi from socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ft559 (ro)
      /var/run/secrets/openshift/serviceaccount from oidc-token (rw)
  objectstorage-provisioner-sidecar:
    Container ID:  cri-o://5bdbf472111c00c3e5573a3da866f58d73ecff4cc3ebefc67be615ca6c298fea
    Image:         gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387
    Image ID:      gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar@sha256:589c0ad4ef5d0855fe487440e634d01315bc3d883f91c44cb72577ea6e12c890
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 03 Jul 2023 12:52:21 -0400
      Finished:     Mon, 03 Jul 2023 12:52:21 -0400
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     100m
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  512Mi
    Environment:
      POD_NAMESPACE:            openshift-storage (v1:metadata.namespace)
      OPERATOR_CONDITION_NAME:  mcg-operator.v4.14.0-36.stable
    Mounts:
      /apiserver.local.config/certificates from apiservice-cert (rw)
      /tmp/k8s-webhook-server/serving-certs from webhook-cert (rw)
      /var/lib/cosi from socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ft559 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  oidc-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3600
  socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  apiservice-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  noobaa-operator-service-cert
    Optional:    false
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  noobaa-operator-service-cert
    Optional:    false
  kube-api-access-ft559:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                             node.ocs.openshift.io/storage=true:NoSchedule
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       3m17s                 default-scheduler  Successfully assigned openshift-storage/noobaa-operator-bf5f597c5-89kzh to lon06-worker-0.rdr-vault414.ibm.com
  Normal   AddedInterface  3m15s                 multus             Add eth0 [10.129.2.111/23] from ovn-kubernetes
  Normal   Pulling         3m15s                 kubelet            Pulling image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726"
  Normal   Pulled          3m10s                 kubelet            Successfully pulled image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726" in 5.412369001s (5.412384842s including waiting)
  Normal   Created         3m9s                  kubelet            Created container noobaa-operator
  Normal   Started         3m9s                  kubelet            Started container noobaa-operator
  Normal   Pulled          3m6s                  kubelet            Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 3.240929206s (3.240946823s including waiting)
  Normal   Pulled          3m5s                  kubelet            Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 409.291437ms (409.307338ms including waiting)
  Normal   Pulled          2m52s                 kubelet            Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 450.403817ms (450.424934ms including waiting)
  Normal   Pulling         2m27s (x4 over 3m9s)  kubelet            Pulling image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387"
  Normal   Pulled          2m27s                 kubelet            Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 431.566038ms (431.598517ms including waiting)
  Normal   Started         2m26s (x4 over 3m6s)  kubelet            Started container objectstorage-provisioner-sidecar
  Warning  BackOff         2m26s (x5 over 3m4s)  kubelet            Back-off restarting failed container objectstorage-provisioner-sidecar in pod noobaa-operator-bf5f597c5-89kzh_openshift-storage(c956cca7-8045-4bea-89aa-73a21665c3b9)
  Normal   Created         2m26s (x4 over 3m6s)  kubelet            Created container objectstorage-provisioner-sidecar

[root@rdr-vault414-lon06-bastion-0 ~]# oc logs pod/noobaa-operator-bf5f597c5-89kzh -n openshift-storage -c objectstorage-provisioner-sidecar
exec /objectstorage-sidecar: exec format error
 
[root@rdr-vault414-lon06-bastion-0 ~]# podman pull gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387
Trying to pull gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387...
Getting image source signatures
Copying blob 2ee0cbba6586 done  
Copying blob 8fdb1fc20e24 done  
Copying config 9e0fc3087a done  
Writing manifest to image destination
Storing signatures
WARNING: image platform (linux/amd64) does not match the expected platform (linux/ppc64le)
9e0fc3087a24a368aa71714243be37b89a74beabdb31f8d0a5c7a655e8702bbb

[root@rdr-vault414-lon06-bastion-0 ~]# podman images
REPOSITORY                                                                  TAG                           IMAGE ID      CREATED       SIZE
gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar  v20221117-v0.1.0-22-g0e67387  9e0fc3087a24  7 months ago  54.2 MB
[root@rdr-vault414-lon06-bastion-0 ~]# podman inspect 9e0fc3087a24 |grep Arch
          "Architecture": "amd64",

Comment 3 Nimrod Becker 2023-07-05 13:30:14 UTC

Talking to Deepshikha Khandelwal, she has confirmed that due to the build outage we were not able to produce a proper d/s build with the new addition (the COSI controller) and suspects this is related.

We expect this to be cleared once the outage is done a new d/s build can be made

Comment 4 Boris Ranto 2023-08-01 13:23:37 UTC

We are overriding this image now so it should be available for all the supported architectures.