Bug 2219436
| Summary: | Unable to deploy ODF4.14 on IBM Power(ppc64le) because of missing multiarch image of objectstorage-provisioner-sidecar in noobaa operator pod | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Aaruni Aggarwal <aaaggarw> |
| Component: | build | Assignee: | Tamil <tmuthami> |
| Status: | CLOSED ERRATA | QA Contact: | Petr Balogh <pbalogh> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.14 | CC: | branto, ebenahar, kramdoss, muagarwa, nbecker, odf-bz-bot |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | ODF 4.14.0 | ||
| Hardware: | ppc64le | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-08 18:52:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Aaruni Aggarwal
2023-07-03 17:05:51 UTC
[root@rdr-vault414-lon06-bastion-0 ~]# oc get csv -A
NAMESPACE NAME DISPLAY VERSION REPLACES PHASE
openshift-local-storage local-storage-operator.v4.14.0-202306270854 Local Storage 4.14.0-202306270854 Succeeded
openshift-operator-lifecycle-manager packageserver Package Server 0.19.0 Succeeded
openshift-storage mcg-operator.v4.14.0-36.stable NooBaa Operator 4.14.0-36.stable Installing
openshift-storage ocs-operator.v4.14.0-36.stable OpenShift Container Storage 4.14.0-36.stable Succeeded
openshift-storage odf-csi-addons-operator.v4.14.0-36.stable CSI Addons 4.14.0-36.stable Succeeded
openshift-storage odf-operator.v4.14.0-36.stable OpenShift Data Foundation 4.14.0-36.stable Succeeded
[root@rdr-vault414-lon06-bastion-0 ~]# oc get pods -n openshift-storage
NAME READY STATUS RESTARTS AGE
csi-addons-controller-manager-6b5b8449f9-gqxvm 2/2 Running 0 2m34s
noobaa-operator-bf5f597c5-89kzh 1/2 CrashLoopBackOff 4 (76s ago) 2m49s
ocs-metrics-exporter-57f48d7867-xv9zn 1/1 Running 0 2m35s
ocs-operator-865d8cb68d-7csfg 1/1 Running 0 2m35s
odf-console-7757bbbc5f-xl8kx 1/1 Running 0 2m53s
odf-operator-controller-manager-5798d84c44-pm8nk 2/2 Running 0 2m53s
rook-ceph-operator-5c555584cd-ncs2m 1/1 Running 0 2m35s
[root@rdr-vault414-lon06-bastion-0 ~]# oc describe pod noobaa-operator-bf5f597c5-89kzh -n openshift-storage
Name: noobaa-operator-bf5f597c5-89kzh
Namespace: openshift-storage
Priority: 0
Service Account: noobaa
Node: lon06-worker-0.rdr-vault414.ibm.com/192.168.0.87
Start Time: Mon, 03 Jul 2023 12:49:27 -0400
Labels: app=noobaa
noobaa-operator=deployment
pod-template-hash=bf5f597c5
Annotations: alm-examples:
[{"kind":"NooBaa","apiVersion":"noobaa.io/v1alpha1","metadata":{"name":"noobaa","creationTimestamp":null},"spec":{"cleanupPolicy":{},"secu...
capabilities: Basic Install
categories: Storage,Big Data
certified: false
containerImage: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726
createdAt: 2019-07-08T13:10:20.940Z
description: NooBaa is an object data service for hybrid and multi cloud environments.
k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.129.2.111/23"],"mac_address":"0a:58:0a:81:02:6f","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.111/2...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"10.129.2.111"
],
"mac": "0a:58:0a:81:02:6f",
"default": true,
"dns": {}
}]
olm.operatorGroup: openshift-storage-5hqns
olm.operatorNamespace: openshift-storage
olm.skipRange: >=4.2.0 <4.14.0-36.stable
olm.targetNamespaces: openshift-storage
olmcahash: 67f52c7ece60d1b142a1ebbe78930d35f9601b77754fc65f1451473c3bd36cf9
openshift.io/scc: restricted-v2
operatorframework.io/properties:
{"properties":[{"type":"olm.gvk","value":{"group":"noobaa.io","kind":"BackingStore","version":"v1alpha1"}},{"type":"olm.gvk","value":{"gro...
operators.openshift.io/infrastructure-features: ֿ'["disconnected"]'
operators.operatorframework.io/operator-type: non-standalone
repository: https://github.com/noobaa/noobaa-operator
seccomp.security.alpha.kubernetes.io/pod: runtime/default
support: Red Hat
Status: Running
IP: 10.129.2.111
IPs:
IP: 10.129.2.111
Controlled By: ReplicaSet/noobaa-operator-bf5f597c5
Containers:
noobaa-operator:
Container ID: cri-o://ea6d8dbf6fa83b99a7a6566df1a68ded4516f35c7e62ff1dd1f6858da32b60a9
Image: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726
Image ID: registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726
Port: <none>
Host Port: <none>
State: Running
Started: Mon, 03 Jul 2023 12:49:35 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 250m
memory: 512Mi
Requests:
cpu: 250m
memory: 512Mi
Environment:
OPERATOR_NAME: noobaa-operator
POD_NAME: noobaa-operator-bf5f597c5-89kzh (v1:metadata.name)
WATCH_NAMESPACE: openshift-storage (v1:metadata.namespace)
NOOBAA_CORE_IMAGE: registry.redhat.io/odf4/mcg-core-rhel9@sha256:791e360f8c0febad70bbde5797908baacb4f1a029bd90fd74734f438fe42b5fa
NOOBAA_DB_IMAGE: registry.redhat.io/rhel8/postgresql-12@sha256:f7f678d44d5934ed3d95c83b4428fee4b616f37e8eadc5049778f133b4ce3713
ENABLE_NOOBAA_ADMISSION: true
OPERATOR_CONDITION_NAME: mcg-operator.v4.14.0-36.stable
Mounts:
/apiserver.local.config/certificates from apiservice-cert (rw)
/tmp/k8s-webhook-server/serving-certs from webhook-cert (rw)
/var/lib/cosi from socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ft559 (ro)
/var/run/secrets/openshift/serviceaccount from oidc-token (rw)
objectstorage-provisioner-sidecar:
Container ID: cri-o://5bdbf472111c00c3e5573a3da866f58d73ecff4cc3ebefc67be615ca6c298fea
Image: gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387
Image ID: gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar@sha256:589c0ad4ef5d0855fe487440e634d01315bc3d883f91c44cb72577ea6e12c890
Port: <none>
Host Port: <none>
Args:
--v=5
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 03 Jul 2023 12:52:21 -0400
Finished: Mon, 03 Jul 2023 12:52:21 -0400
Ready: False
Restart Count: 5
Limits:
cpu: 100m
memory: 512Mi
Requests:
cpu: 100m
memory: 512Mi
Environment:
POD_NAMESPACE: openshift-storage (v1:metadata.namespace)
OPERATOR_CONDITION_NAME: mcg-operator.v4.14.0-36.stable
Mounts:
/apiserver.local.config/certificates from apiservice-cert (rw)
/tmp/k8s-webhook-server/serving-certs from webhook-cert (rw)
/var/lib/cosi from socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ft559 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
oidc-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
apiservice-cert:
Type: Secret (a volume populated by a Secret)
SecretName: noobaa-operator-service-cert
Optional: false
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: noobaa-operator-service-cert
Optional: false
kube-api-access-ft559:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
node.ocs.openshift.io/storage=true:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m17s default-scheduler Successfully assigned openshift-storage/noobaa-operator-bf5f597c5-89kzh to lon06-worker-0.rdr-vault414.ibm.com
Normal AddedInterface 3m15s multus Add eth0 [10.129.2.111/23] from ovn-kubernetes
Normal Pulling 3m15s kubelet Pulling image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726"
Normal Pulled 3m10s kubelet Successfully pulled image "registry.redhat.io/odf4/mcg-rhel9-operator@sha256:17457179864ee10f6dc197a72fc8ea3edb21c952bcd23c074f59609d6c487726" in 5.412369001s (5.412384842s including waiting)
Normal Created 3m9s kubelet Created container noobaa-operator
Normal Started 3m9s kubelet Started container noobaa-operator
Normal Pulled 3m6s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 3.240929206s (3.240946823s including waiting)
Normal Pulled 3m5s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 409.291437ms (409.307338ms including waiting)
Normal Pulled 2m52s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 450.403817ms (450.424934ms including waiting)
Normal Pulling 2m27s (x4 over 3m9s) kubelet Pulling image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387"
Normal Pulled 2m27s kubelet Successfully pulled image "gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387" in 431.566038ms (431.598517ms including waiting)
Normal Started 2m26s (x4 over 3m6s) kubelet Started container objectstorage-provisioner-sidecar
Warning BackOff 2m26s (x5 over 3m4s) kubelet Back-off restarting failed container objectstorage-provisioner-sidecar in pod noobaa-operator-bf5f597c5-89kzh_openshift-storage(c956cca7-8045-4bea-89aa-73a21665c3b9)
Normal Created 2m26s (x4 over 3m6s) kubelet Created container objectstorage-provisioner-sidecar
[root@rdr-vault414-lon06-bastion-0 ~]# oc logs pod/noobaa-operator-bf5f597c5-89kzh -n openshift-storage -c objectstorage-provisioner-sidecar
exec /objectstorage-sidecar: exec format error
[root@rdr-vault414-lon06-bastion-0 ~]# podman pull gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387
Trying to pull gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20221117-v0.1.0-22-g0e67387...
Getting image source signatures
Copying blob 2ee0cbba6586 done
Copying blob 8fdb1fc20e24 done
Copying config 9e0fc3087a done
Writing manifest to image destination
Storing signatures
WARNING: image platform (linux/amd64) does not match the expected platform (linux/ppc64le)
9e0fc3087a24a368aa71714243be37b89a74beabdb31f8d0a5c7a655e8702bbb
[root@rdr-vault414-lon06-bastion-0 ~]# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar v20221117-v0.1.0-22-g0e67387 9e0fc3087a24 7 months ago 54.2 MB
[root@rdr-vault414-lon06-bastion-0 ~]# podman inspect 9e0fc3087a24 |grep Arch
"Architecture": "amd64",
Talking to Deepshikha Khandelwal, she has confirmed that due to the build outage we were not able to produce a proper d/s build with the new addition (the COSI controller) and suspects this is related. We expect this to be cleared once the outage is done a new d/s build can be made We are overriding this image now so it should be available for all the supported architectures. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6832 |