Description of problem: The oc binary for mac arm64 can’t be executed Version-Release number of selected component (if applicable): openshift-client-mac-arm64-4.10.0-rc.6.tar.gz How reproducible: always Steps to Reproduce: 1.test@NoradeMacBook-Pro Public % arch arm64 2. Download the binary oc for mac-arm64: test@NoradeMacBook-Pro Public % ls -tl total 475456 -rwxr-xr-x@ 1 test staff 40855414 Feb 28 16:42 openshift-client-mac-arm64-4.10.0-rc.6.tar.gz 3. Unzip it and try to execute Actual results: 3. Can’t execute the oc binary for mac-arm64 : tar xzvf openshift-client-mac-arm64-4.10.0-rc.6.tar.gz x README.md x oc x kubectl test@NoradeMacBook-Pro Public % ./oc version --client zsh: killed ./oc version --client Expected results: 3. Could run the oc binary for mac-arm64 Additional info:
I investigated this in https://github.com/openshift/oc/issues/1221#issuecomment-1231651942 , it's caused by the version patching that 'oc adm release extract' is doing, which invalidates the binaries on darwin/arm64 (macOS detects them as corrupt). This also impacts openshift-install. A workaround would be to disable the version patching on darwin/arm64, but then `oc version` will be confusing.
For crc, we need a working native oc binary that we can use on macOS/m1, so it would be really useful if this bug got some attention. And it's sad that we've been shipping crashing binaries on mirror.openshift.com for 6+ months :(
All macOS M1 binaries have an 'adhoc' signature, see https://github.com/golang/go/issues/42684 If we binary patch oc executable without changing the signature, it's expected the binary won't load anymore. Running 'codesign --force -s - oc' after downloading/unpacking it allows the binary to start.
Hi Christophe, has changing the signature become required after patching a binary recently on Mac? Ying, is 4.10 the first ocp version that reproduces the issue? Is the latest 4.9 binary functioning as expected?
(In reply to Jan Chaloupka from comment #4) > Hi Christophe, has changing the signature become required after patching a > binary recently on Mac? The golang issue I linked to says that binary signing is required for M1 binaries, so my guess would be that all oc/openshift-install M1 binaries on mirror.openshift.com are impacted.
(In reply to Jan Chaloupka from comment #4) > Ying, is 4.10 the first ocp version that reproduces the issue? Is the latest > 4.9 binary functioning as expected? I can't see arm64 binaries on https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest-4.9/ oc-macos-arm64-4.10.0 shows this issue.
I'm going to start investigate the issue but first I'd like to ask that have you ever tried with latest oc-macos-arm64 version?, I wonder problem has been fixed already or still persisting also in newer releases?
I did the investigation in https://github.com/openshift/oc/issues/1221#issuecomment-1231651942 with a 4.11 binary. I just tried again with something newer: ``` % curl -O -L https://mirror.openshift.com/pub/openshift-v4/aarch64/clients/ocp-dev-preview/latest-4.12/openshift-client-mac-arm64-4.12.0-ec.2.tar.gz % tar xvf ./openshift-client-mac-arm64-4.12.0-ec.2.tar.gz x README.md x oc x kubectl % ./oc zsh: killed ./oc ``` pkg/cli/admin/release/extract_tools.go still has the problematic code, and looking at pkg/cli/admin/release/ git history, I don't see changes which would help with the invalid signature.
"the problematic code" does version injection and can not be removed in pkg/cli/admin/release/extract_tools.go. Because that causes version command will not return correct value and this will be another problem. As far as I can see in here https://github.com/openshift/oc/issues/1221#issuecomment-1233992972, you said `codesign --force -s - oc` fixes problem. Maybe we can try to add this somewhere for the macos binaries?
The problem is that codesign is a macOS-only binary and I don't think you are doing the binary extraction on a mac? golang manages to do this adhoc codesign in pure go code, but I haven't read the whole issue ( https://github.com/golang/go/issues/42684 ) so I have no idea if there is code which can be reused, or even a go binary doing the same job as `codesign --force -s - oc` In an ideal world, the macOS binaries would be signed by Red Hat, and notarized, ... which would workaround this bug if the signing is done between the binary extraction and the upload to the mirror, but as far as I know there is no automated process in place for signing of macOS binaries :-/
Thanks Christophe for helping me while investigated the issue. Problem is exactly like sorted out in here https://github.com/openshift/oc/issues/1221#issuecomment-1231651942 and only happening oc-mac-arm64. Since we can not change version patching in short term, we need to move forward with three different workarounds; 1) $brew install openshift-cli this installs working oc binary. Only drawback is binary returns incorrect versions(because it looks that binary is not patched with versions). 2) Using x86_64 binary which already works as expected. 3) User manually signs oc-mac-arm64 binary by executing `codesign -f -s -` In the manner of this bug, we might want to add manually signing step into our docs if user gets such an error. Other than that, I'd prefer closing this bug as wontfix.
> and only happening oc-mac-arm64. This also happens for the mac-arm64 versions of openshift-install. > Since we can not change version patching in short term, we need to move forward with three different workarounds; [...] In the manner of this bug, we might want to add manually signing step into our docs if user gets such an error. Other than that, I'd prefer closing this bug as wontfix. From a customer perspective, if I go to mirror.openshift.com from my mac M1 and download oc or openshift-install from there, if the binary does not work out of the box but crashes, this makes us (Red Hat) look very bad, especially when this happens in all releases, and is a known issue. If we don't have time short term for a real fix, we should either: - remove binary patching when extracting arm64 binaries, versioning will be wrong, but at least the binaries will be working - remove the arm64 binaries altogether from mirror.openshift.com, customers will use the amd64 binaries instead, and they won't see a crash
A good workaround to avoid this bug would be to add a Red Hat signature to the binary after it's extracted but before it's uploaded. This would make the binary more trustable for the customer as the signature would guarantee it's coming from Red Hat, and the RH signature process would put back a valid signature for the oc binary. The downside to that is that this adds an additional step to the release process, and this can be a bit complicated to setup as this needs manual intervention from the EXD team at the moment.
The code the go compiler/linker uses for codesigning is just one file https://github.com/golang/go/blob/master/src/cmd/internal/codesign/codesign.go `go tool buildid` has code to regenerate/rewrite the signature when it changes https://github.com/golang/go/commit/8cd35e00bd413e68e6b9ae6403aa4209fc89b90f#diff-7394c00849e0b64c954228e0e8cde79c3816ed9256a32ffc9993ca088bf92750 we could most likely reuse that code in `oc adm release extract`
We had a discussions about removal of version injection for mac os arm64 on https://github.com/openshift/oc/pull/1233 and we concluded that this is not preferable way. Another thing is that binaries downloaded from web console should work without any problem.
And lastly, if it works for you, we can add a warning message in `oc adm release extract` by guiding the user which commands might be needed to run oc and openshift-install binaries.
> Another thing is that binaries downloaded from web console should work without any problem. When I tried the binary from the web console, they were not working, but it seemed to be a different problem than the one discussed here, I did not spend much time on testing this. > And lastly, if it works for you, we can add a warning message in `oc adm release extract` by guiding the user which commands might be needed to run oc and openshift-install binaries. The user is not running `oc adm release extract`, they are just trying to get an oc/openshift-install binary from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.11.1/openshift-client-mac-arm64-4.11.1.tar.gz
(In reply to Jan Chaloupka from comment #4) > Hi Christophe, has changing the signature become required after patching a > binary recently on Mac? > > Ying, is 4.10 the first ocp version that reproduces the issue? Is the latest > 4.9 binary functioning as expected? Hi Jan, 4.10 release is where we started off with the arm based oc client and we did hit this issue with this release itself. Please check the thread here for more info with maciej. https://coreos.slack.com/archives/GK58XC2G2/p1646038857473939 Thanks kasturi
Hello, Downloading oc package from webconsole as described at [1] and step 3 changes a little bit for this specific case here i.e download mac for arm64 platform , it works fine and i do not see any issues with that. Below are the steps i have followed to verify the same. ➜ Public ./oc version Client Version: 4.10.0-202208301216.p0.gdd2bcd0.assembly.stream-dd2bcd0 Server Version: 4.10.31 Kubernetes Version: v1.23.5+012e945 1. Installed openshift cluster with version 4.10.31 2. Browse through the console url 3. From the webconsole click '?' 4. click Command Line Tools 5. Select the oc binary for mac for amr64 platform 6. save the file 7. copied it over to the mac machine 7. unpack and unzip the archive 8. Move the oc binary to a directory on your PATH [1] https://docs.openshift.com/container-platform/4.11/cli_reference/openshift_cli/getting-started-cli.html#cli-installing-cli-web-console-macos_cli-developer-commands Adding my test steps here which passed with out any issues: ============================================================= ➜ Public arch arm64 ➜ Public export KUBECONFIG=kubeconfig_41031 ➜ Public ./oc version Client Version: 4.10.0-202208301216.p0.gdd2bcd0.assembly.stream-dd2bcd0 Server Version: 4.10.31 Kubernetes Version: v1.23.5+012e945 ➜ Public ./oc login -u kubeadmin -p <pwd> Login successful. You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". ➜ Public ./oc get nodes ➜ Public ./oc adm cordon <node_name> ➜ Public ./oc adm top node ➜ Public ./oc new-project knarra ➜ Public ./oc get pods ➜ Public ./oc rsh postgresql-1-4mc6w sh-4.4$ exit exit ➜ Public ./oc exec postgresql-1-4mc6w -- ls -l total 0 ➜ Public ./oc logs -f postgresql-1-deploy --> Scaling postgresql-1 to 1 --> Success ➜ Public ./oc debug node/<node> Starting pod/ip-<node>-debug ... To use host binaries, run `chroot /host` chrootPod IP: <IP> If you don't see a command prompt, try pressing enter. chroot /host sh-4.4# pwd / sh-4.4# exit exit sh-4.4# exit ➜ Public ./oc explain pod KIND: Pod VERSION: v1 DESCRIPTION: Pod is a collection of containers that can run on a host. This resource is created by clients and scheduled onto hosts. ➜ Public ./oc help OpenShift Client This client helps you develop, build, deploy, and run your applications on any OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the 'adm' subcommand. Usage: oc [flags] ➜ Public ./oc create dc hello --image=<image> deploymentconfig.apps.openshift.io/hello created ➜ Public ./oc create deployment kuard --image gcr.io/kuar-demo/kuard-amd64:blue deployment.apps/kuard created ➜ Public ./oc get dc NAME REVISION DESIRED CURRENT TRIGGERED BY hello 1 1 1 config postgresql 1 1 1 config,image(postgresql:12-el8) rails-postgresql-example 1 1 1 config,image(rails-postgresql-example:latest) ➜ Public ./oc delete deployment kuard deployment.apps "kuard" deleted ➜ Public ./oc delete dc hello deploymentconfig.apps.openshift.io "hello" deleted ➜ Public ./oc get deployment No resources found in knarra namespace. ➜ Public ./oc api-versions admissionregistration.k8s.io/v1 apiextensions.k8s.io/v1 apiregistration.k8s.io/v1 apiserver.openshift.io/v1 ➜ Public ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.31 True False 29m Cluster version is 4.10.31 Public ./oc adm must-gather [must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8462387895747096e0aadaea8ab19129fdf19604a1a2e98698873762f73c1799 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 49df802a-ff9d-4830-8b71-83a8d599b669 ClusterVersion: Stable at "4.10.31" ClusterOperators: All healthy and stable Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 49df802a-ff9d-4830-8b71-83a8d599b669 ClusterVersion: Stable at "4.10.31" ClusterOperators: All healthy and stable ➜ Public ls -l drwxr-xr-x 5 huangmingxia staff 160 Sep 12 19:24 must-gather.local.4776577139288174794 ➜ Public ./oc adm inspect etcd Wrote inspect data to inspect.local.674946650715380617. ➜ Public ls -l inspect.local.674946650715380617 total 16 drwxr-xr-x 3 huangmingxia staff 96 Sep 12 19:25 cluster-scoped-resources -rw-r--r-- 1 huangmingxia staff 3336 Sep 12 19:25 event-filter.html -rw-r--r-- 1 huangmingxia staff 104 Sep 12 19:25 timestamp ➜ Public ./oc logout -> Logged out as admin user Logged "kube:admin" out on "https://api.knarra-09123.qe.devcluster.openshift.com:6443" ➜ Public ./oc login -u testuser-2 -p <pwd> ---------> Logged in as normal user Login successful. You don't have any projects. You can try to create a new project, by running oc new-project <projectname> ➜ Public ./oc new-project knarra2 Now using project "knarra2" on server "https://api.knarra-09123.qe.devcluster.openshift.com:6443". Another way i used to download to mac instance is by using the way below: ========================================================================= 1. curl -k -o openshiftmac.zip https://<url>/arm64/mac/oc.zip 2. unzip openshiftmac.zip Archive: openshiftmac.zip extracting: oc ➜ Public ./oc version Client Version: 4.10.0-202208301216.p0.gdd2bcd0.assembly.stream-dd2bcd0 Kubernetes Version: v1.23.5+012e945 Hope this helps. Please let us know if you are facing any issues with the above method. Meanwhile i will also open a doc bug to get this added to the docs, thanks !!
Thank you Kasturi. I was also able to successfully install `oc` client by downloading it from console.redhat.com. The ARM64 macOS binary is not available there, so I didn't have chance to . I however had to go the the steps of approving a binary from unknown developer (https://support.apple.com/en-eg/guide/mac-help/mh40616/mac). This is a different problem than described in the issue, however I guess it could be fixed within the same productisation pipeline. If ARM64 macOS binaries could be removed altogether from mirror.openshift.com, it looks like problem will be much less exposed. Would the be possible? I would still consider our inability to build native macOS binary unfortunate (and also you should consider that after 2+ years on the market people will start uninstalling Rosetta 2 emulation layer or it might not be enable by default in future macOS versions) Christophe, `kubectl` binary distributed in our ARM64 Mac bundle is also broken, the same way as `oc`. This is weird, as official `kubectl` binary (https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/) works as expected (known developer, signed, native binary for M1). Is there a reason why kubectl binary is modified? Thanks, Karel
(In reply to Karel Piwko from comment #21) > Thank you Kasturi. > > I was also able to successfully install `oc` client by downloading it from > console.redhat.com. Not sure if that was the right way to do it. I followed steps here in the doc [1] and installed directly. The console i logged onto was the openshift web console, which i retrieved using from a openshift cluster by running 'oc whoami --show-console'. https://docs.openshift.com/container-platform/4.11/cli_reference/openshift_cli/getting-started-cli.html#cli-installing-cli-web-console-macos_cli-developer-commands The ARM64 macOS binary is not available there, so I > didn't have chance to . I however had to go the the steps of approving a > binary from unknown developer > (https://support.apple.com/en-eg/guide/mac-help/mh40616/mac). > This is a different problem than described in the issue, however I guess it > could be fixed within the same productisation pipeline. > > If ARM64 macOS binaries could be removed altogether from > mirror.openshift.com, it looks like problem will be much less exposed. Would > the be possible? > > I would still consider our inability to build native macOS binary > unfortunate (and also you should consider that after 2+ years on the market > people will start uninstalling Rosetta 2 emulation layer or it might not be > enable by default in future macOS versions) > > Christophe, `kubectl` binary distributed in our ARM64 Mac bundle is also > broken, the same way as `oc`. This is weird, as official `kubectl` binary > (https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/) works as > expected (known developer, signed, native binary for M1). Is there a reason > why kubectl binary is modified? > > Thanks, > > Karel
The problem I have with this flow is that it requires an OpenShift instance running. I don't have that. I have downloaded it from OSD instance I have access to and I can confirm it behaves the same as the one from console.redhat.com/openshift/downloads. (Note I don't know what version that is, `oc version` says *Kubernetes Version: v1.23.5+012e945* and `oc get clusterversion` says permission denied. That said, there is only x86_64 version and after allowing a binary downloaded from internet, it works as expected. (In reply to RamaKasturi from comment #22) > (In reply to Karel Piwko from comment #21) > > Thank you Kasturi. > > > > I was also able to successfully install `oc` client by downloading it from > > console.redhat.com. > > Not sure if that was the right way to do it. I followed steps here in the > doc [1] and installed directly. The console i logged onto was the openshift > web console, which i retrieved using from a openshift cluster by running 'oc > whoami --show-console'. > https://docs.openshift.com/container-platform/4.11/cli_reference/ > openshift_cli/getting-started-cli.html#cli-installing-cli-web-console- > macos_cli-developer-commands > > The ARM64 macOS binary is not available there, so I > > didn't have chance to . I however had to go the the steps of approving a > > binary from unknown developer > > (https://support.apple.com/en-eg/guide/mac-help/mh40616/mac). > > This is a different problem than described in the issue, however I guess it > > could be fixed within the same productisation pipeline. > > > > If ARM64 macOS binaries could be removed altogether from > > mirror.openshift.com, it looks like problem will be much less exposed. Would > > the be possible? > > > > I would still consider our inability to build native macOS binary > > unfortunate (and also you should consider that after 2+ years on the market > > people will start uninstalling Rosetta 2 emulation layer or it might not be > > enable by default in future macOS versions) > > > > Christophe, `kubectl` binary distributed in our ARM64 Mac bundle is also > > broken, the same way as `oc`. This is weird, as official `kubectl` binary > > (https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/) works as > > expected (known developer, signed, native binary for M1). Is there a reason > > why kubectl binary is modified? > > > > Thanks, > > > > Karel
(In reply to Karel Piwko from comment #23) > The problem I have with this flow is that it requires an OpenShift instance > running. I don't have that. > I have downloaded it from OSD instance I have access to and I can confirm it > behaves the same as the one from console.redhat.com/openshift/downloads. > > (Note I don't know what version that is, `oc version` says `4.10.0-202206270836.p0.g45460a5.assembly.stream-45460a5` I'll try it with a newer version. > > (In reply to RamaKasturi from comment #22) > > (In reply to Karel Piwko from comment #21) > > > Thank you Kasturi. > > > > > > I was also able to successfully install `oc` client by downloading it from > > > console.redhat.com. > > > > Not sure if that was the right way to do it. I followed steps here in the > > doc [1] and installed directly. The console i logged onto was the openshift > > web console, which i retrieved using from a openshift cluster by running 'oc > > whoami --show-console'. > > https://docs.openshift.com/container-platform/4.11/cli_reference/ > > openshift_cli/getting-started-cli.html#cli-installing-cli-web-console- > > macos_cli-developer-commands > > > > The ARM64 macOS binary is not available there, so I > > > didn't have chance to . I however had to go the the steps of approving a > > > binary from unknown developer > > > (https://support.apple.com/en-eg/guide/mac-help/mh40616/mac). > > > This is a different problem than described in the issue, however I guess it > > > could be fixed within the same productisation pipeline. > > > > > > If ARM64 macOS binaries could be removed altogether from > > > mirror.openshift.com, it looks like problem will be much less exposed. Would > > > the be possible? > > > > > > I would still consider our inability to build native macOS binary > > > unfortunate (and also you should consider that after 2+ years on the market > > > people will start uninstalling Rosetta 2 emulation layer or it might not be > > > enable by default in future macOS versions) > > > > > > Christophe, `kubectl` binary distributed in our ARM64 Mac bundle is also > > > broken, the same way as `oc`. This is weird, as official `kubectl` binary > > > (https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/) works as > > > expected (known developer, signed, native binary for M1). Is there a reason > > > why kubectl binary is modified? > > > > > > Thanks, > > > > > > Karel
Customers following https://access.redhat.com/solutions/6964441 would hit this bug.
Just wanted to confirm that on a newer OpenShift cluster that Kasturi provided to me, I was able to download an oc binary that is Apple ARM64 native: ➜ from-ocp-kasturi file oc oc: Mach-O 64-bit executable arm64 ➜ from-ocp-kasturi ./oc version Client Version: 4.10.0-202208301216.p0.gdd2bcd0.assembly.stream-dd2bcd0 I still had to confirm unknown developer and allow executing a binary from the internet, yet it was not broken.
can't reproduce the issue now : ec2-user@ip-172-31-113-190 ~ % tar xzvf openshift-client-mac-arm64-4.12.0-ec.4.tar.gz x README.md x oc x kubectl ec2-user@ip-172-31-113-190 ~ % ./oc version Client Version: 4.12.0-ec.4 Kustomize Version: v4.5.4
Thank you Kasturi. I can confirm that the problem with broken binary on M1 has been fixed. There is still the problem of unknown developer (I had to go the the steps of approving a binary from unknown developer (https://support.apple.com/en-eg/guide/mac-help/mh40616/mac). This is a different problem than described in the issue, however I guess it could be fixed within the same productisation pipeline.) Should this be tracked as a different issue?
@arda any idea about the above ?
(In reply to Karel Piwko from comment #36) > Thank you Kasturi. > > I can confirm that the problem with broken binary on M1 has been fixed. > > > There is still the problem of unknown developer (I had to go the the steps > of approving a binary from unknown developer > (https://support.apple.com/en-eg/guide/mac-help/mh40616/mac). > This is a different problem than described in the issue, however I guess it > could be fixed within the same productisation pipeline.) > > Should this be tracked as a different issue? Hi, We fixed that problem in oc by mimicking a workaround applied in Go itself. Because it is affecting users on that architectures. However, unknown developer problem, as you said, is a different problem and can not be fixed in oc. I think, our current overall architecture does not support it at least for now.
Ack, thanks Arda.
https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/4.12.0-ec.4/openshift-client-mac-arm64-4.12.0-ec.4.tar.gz is still broken, but I think it's because it was uploaded before the bug was fixed. It's quite confusing though that it's possible to download a different openshift-client-mac-arm64-4.12.0-ec.4.tar.gz tarball which is working.
@christophe Fergeau Please allow us some time and we will check and get back. @yingzhou could you please help downlaod the binary from the above link and see if that is still broken for you ? Thanks !!
@christophe Fergeau i just spoke to the dev team and they were also of the similar opinion as you, one suggestion was to wait till ec5 is out and gets uploaded to the link in Comment 40. We will test ec5 when it is out and update the bug with details. Thanks !! More details on the discussion can be found here at [1] @yingzhou let us test the bug by downloading ec5 bits from https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/ when the bits are available there. [1] https://coreos.slack.com/archives/GK58XC2G2/p1666273718028049
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399