Bug 2193114
| Summary: | MCG replication is failing due to OC binary incompatible on Power platform | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | narayanspg <ngowda> |
| Component: | build | Assignee: | Boris Ranto <branto> |
| Status: | CLOSED ERRATA | QA Contact: | Petr Balogh <pbalogh> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.13 | CC: | branto, muagarwa, ocs-bugs, odf-bz-bot |
| Target Milestone: | --- | ||
| Target Release: | ODF 4.13.0 | ||
| Hardware: | ppc64le | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 4.13.0-201 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-06-21 15:25:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
workaround test: Tried to test by manual copy of Power compatible binaries onto the noobaa-core-0 pod. copied the Power compatible binaries of archived oc and kubectl onto the tmp folder of noobaa-core-0 pod. But not able to overwrite at the binaries location due to root or sudo permission required. sh-5.1$ cp /tmp/usr/local/bin/oc /usr/bin/oc cp: cannot create regular file '/usr/bin/oc': Permission denied Issue Impact:
due the oc binary compatibility issue, MCG replication feature is not working.
created namespace stores, Object classess and OBC's. created replication policy with below configurations in OBC for source and target.
Source OBC yaml:
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: source-odf
namespace: openshift-storage
spec:
additionalConfig:
bucketclass: sourcebucketclass
replicationPolicy: '{"rules": [{"rule_id": "example_rule_id", "destination_bucket":"target-odf",
"sync_deletions": true}],"log_replication_info":{"logs_location":{"logs_bucket":
"logs-odf","prefix":""}}}'
storageClassName: openshift-storage.noobaa.io
bucketName: source-odf
target OBC yaml:
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: target-odf
namespace: openshift-storage
spec:
additionalConfig:
bucketclass: targetbucketclass
storageClassName: openshift-storage.noobaa.io
bucketName: target-odf
When we create objects in the source bucket they are not getting replicated on the target bucket.
#Noobaa core logs
May-4 11:16:30.047 [BGWorkers/34] [WARN] core.util.os_utils:: discover_k8s_services: could not list k8s services: Error: Command failed: kubectl get service --selector="app=noobaa" -o=json
/bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error
at ChildProcess.exithandler (node:child_process:412:12)
at ChildProcess.emit (node:events:513:28)
at ChildProcess.emit (node:domain:489:12)
at maybeClose (node:internal/child_process:1091:16)
at Socket.<anonymous> (node:internal/child_process:449:11)
at Socket.emit (node:events:513:28)
at Socket.emit (node:domain:489:12)
at Pipe.<anonymous> (node:net:313:12) {
code: 126,
killed: false,
signal: null,
cmd: 'kubectl get service --selector="app=noobaa" -o=json',
stdout: '',
stderr: '/bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error\n'
}
I believe this should help: https://gitlab.cee.redhat.com/ceph/rhodf/-/merge_requests/1364 For some reason, the image we used before had x86_64 oc/kubectl binaries for ppc64le in it. The new released image should have it fixed. Hi , I have tested the fix and working fine. Power binaries are present in the noobaa core and MCG replication is working. Thanks for the quick fix. you can close this case. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:3742 |
Description of problem (please be detailed as possible and provide log snippests): OC/Kubectl binary present in the noobaa are not compatible on Powerplatform. due to this MCG bucket replication is not working. Version of all relevant components (if applicable): [root@nara-mcg1-odf-693c-sao01-bastion-0 ~]# oc describe csv odf-operator.v4.13.0 -n openshift-storage | grep full Labels: full_version=4.13.0-182 f:full_version: Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? MCG bucket replication feature is not working on Power platform Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 3 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: NO Steps to Reproduce: 1. create OCP cluster 2. deploy ODF 3. check the oc/kubectl binaries inside the noobaa which are not power based. 4. due to this MCG replication is not happening from source to target buckets. Actual results: MCG replication - replicating from source to target bucket should work fine. Expected results: MCG replication should work fine replicating source to target bucket. Additional info: ClusterID: 185491bf-acea-446c-9282-c942f5c09556 ClusterVersion: Stable at "4.13.0-0.nightly-ppc64le-2023-05-02-182828" ClusterOperators: All healthy and stable [root@nara-mcg1-odf-693c-sao01-bastion-0 ~]# oc rsh noobaa-core-0 sh-5.1$ oc sh: /usr/bin/oc: cannot execute binary file: Exec format error sh-5.1$ sh-5.1$ sh-5.1$ kubectl sh: /usr/local/bin/kubectl: cannot execute binary file: Exec format error sh-5.1$ #Noobaa core logs May-4 11:16:30.047 [BGWorkers/34] [WARN] core.util.os_utils:: discover_k8s_services: could not list k8s services: Error: Command failed: kubectl get service --selector="app=noobaa" -o=json /bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error at ChildProcess.exithandler (node:child_process:412:12) at ChildProcess.emit (node:events:513:28) at ChildProcess.emit (node:domain:489:12) at maybeClose (node:internal/child_process:1091:16) at Socket.<anonymous> (node:internal/child_process:449:11) at Socket.emit (node:events:513:28) at Socket.emit (node:domain:489:12) at Pipe.<anonymous> (node:net:313:12) { code: 126, killed: false, signal: null, cmd: 'kubectl get service --selector="app=noobaa" -o=json', stdout: '', stderr: '/bin/sh: line 1: /usr/local/bin/kubectl: cannot execute binary file: Exec format error\n' }