Description of problem (please be detailed as possible and provide log snippests): OC/Kubectl binary present in the noobaa are not compatible on Powerplatform. due to this MCG bucket replication is not working. Version of all relevant components (if applicable): [root@nara-mcg1-odf-693c-sao01-bastion-0 ~]# oc describe csv odf-operator.v4.13.0 -n openshift-storage | grep full Labels: full_version=4.13.0-182 f:full_version: Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? MCG bucket replication feature is not working on Power platform Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 3 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: NO Steps to Reproduce: 1. create OCP cluster 2. deploy ODF 3. check the oc/kubectl binaries inside the noobaa which are not power based. 4. due to this MCG replication is not happening from source to target buckets. Actual results: MCG replication - replicating from source to target bucket should work fine. Expected results: MCG replication should work fine replicating source to target bucket. Additional info: ClusterID: 185491bf-acea-446c-9282-c942f5c09556 ClusterVersion: Stable at "4.13.0-0.nightly-ppc64le-2023-05-02-182828" ClusterOperators: All healthy and stable [root@nara-mcg1-odf-693c-sao01-bastion-0 ~]# oc rsh noobaa-core-0 sh-5.1$ oc sh: /usr/bin/oc: cannot execute binary file: Exec format error sh-5.1$ sh-5.1$ sh-5.1$ kubectl sh: /usr/local/bin/kubectl: cannot execute binary file: Exec format error sh-5.1$ #Noobaa core logs May-4 11:16:30.047 [BGWorkers/34] [WARN] core.util.os_utils:: discover_k8s_services: could not list k8s services: Error: Command failed: kubectl get service --selector="app=noobaa" -o=json /bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error at ChildProcess.exithandler (node:child_process:412:12) at ChildProcess.emit (node:events:513:28) at ChildProcess.emit (node:domain:489:12) at maybeClose (node:internal/child_process:1091:16) at Socket.<anonymous> (node:internal/child_process:449:11) at Socket.emit (node:events:513:28) at Socket.emit (node:domain:489:12) at Pipe.<anonymous> (node:net:313:12) { code: 126, killed: false, signal: null, cmd: 'kubectl get service --selector="app=noobaa" -o=json', stdout: '', stderr: '/bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error\n' }
workaround test: Tried to test by manual copy of Power compatible binaries onto the noobaa-core-0 pod. copied the Power compatible binaries of archived oc and kubectl onto the tmp folder of noobaa-core-0 pod. But not able to overwrite at the binaries location due to root or sudo permission required. sh-5.1$ cp /tmp/usr/local/bin/oc /usr/bin/oc cp: cannot create regular file '/usr/bin/oc': Permission denied
Issue Impact: due the oc binary compatibility issue, MCG replication feature is not working. created namespace stores, Object classess and OBC's. created replication policy with below configurations in OBC for source and target. Source OBC yaml: apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: source-odf namespace: openshift-storage spec: additionalConfig: bucketclass: sourcebucketclass replicationPolicy: '{"rules": [{"rule_id": "example_rule_id", "destination_bucket":"target-odf", "sync_deletions": true}],"log_replication_info":{"logs_location":{"logs_bucket": "logs-odf","prefix":""}}}' storageClassName: openshift-storage.noobaa.io bucketName: source-odf target OBC yaml: apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: target-odf namespace: openshift-storage spec: additionalConfig: bucketclass: targetbucketclass storageClassName: openshift-storage.noobaa.io bucketName: target-odf When we create objects in the source bucket they are not getting replicated on the target bucket. #Noobaa core logs May-4 11:16:30.047 [BGWorkers/34] [WARN] core.util.os_utils:: discover_k8s_services: could not list k8s services: Error: Command failed: kubectl get service --selector="app=noobaa" -o=json /bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error at ChildProcess.exithandler (node:child_process:412:12) at ChildProcess.emit (node:events:513:28) at ChildProcess.emit (node:domain:489:12) at maybeClose (node:internal/child_process:1091:16) at Socket.<anonymous> (node:internal/child_process:449:11) at Socket.emit (node:events:513:28) at Socket.emit (node:domain:489:12) at Pipe.<anonymous> (node:net:313:12) { code: 126, killed: false, signal: null, cmd: 'kubectl get service --selector="app=noobaa" -o=json', stdout: '', stderr: '/bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error\n' }
I believe this should help: https://gitlab.cee.redhat.com/ceph/rhodf/-/merge_requests/1364 For some reason, the image we used before had x86_64 oc/kubectl binaries for ppc64le in it. The new released image should have it fixed.
Hi , I have tested the fix and working fine. Power binaries are present in the noobaa core and MCG replication is working. Thanks for the quick fix. you can close this case.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:3742