Version: $ openshift-install version 4.8 Platform: #Please specify the platform type: aws, libvirt, openstack or baremetal etc. Please specify: * IPI (automated install with `openshift-install`. If you don't know, then it's IPI) * UPI (semi-manual installation on customized infrastructure) What happened? 4.8 builds of ose-installer are failing for ppc64le: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36401931 error: + go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=v4.8.0 -X github.com/openshift/installer/pkg/version.Commit=18974847b6a68fef21f3fae7320259f4ed3beff8 -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install # github.com/openshift/installer/cmd/openshift-install github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingRootPasswordRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingRootPasswordRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsService).HostService: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsService).HostService: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*HostsService).String: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking /usr/lib/golang/pkg/tool/linux_ppc64le/link: too many errors running 'hack/build.sh' failed with exit code 2 What did you expect to happen? A working build How to reproduce it (as minimally and precisely as possible)? $ your-commands-here Anything else we need to know? #Enter text here.
This looks related: https://github.com/golang/go/issues/45564
[QA Summary] I can see that the version "4.8.0-0.nightly-2021-04-29-071720" was "Accepted" 7 hours ago and also published as dev-preview here[1], this version contains the expected PR[2], and therefore also the pending ones since previous commit "fdf0269c3": ~~~ $ oc adm release info --commits registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-29-071720 --registry-config=pull-secret-ci | grep ' installer ' installer https://github.com/openshift/installer 52fef9e1d7b765470e493670177eefd73bc27c6f $ git show 52fef9e1d7b765470e493670177eefd73bc27c6f commit 52fef9e1d7b765470e493670177eefd73bc27c6f Merge: 911eb7f69 24ac0a15e Author: OpenShift Merge Robot <openshift-merge-robot.github.com> Date: Wed Apr 28 18:55:07 2021 -0400 Merge pull request #4875 from multi-arch/ppc64le-link Bug 1953551: Link ppc64le binary dynamically $ git --no-pager log --oneline --first-parent 52fef9e1d -3 52fef9e1d Merge pull request #4875 from multi-arch/ppc64le-link 911eb7f69 Merge pull request #4868 from wking/stable-4.8 18974847b Merge pull request #4755 from staebler/user_tags_in_cluster ~~~ [1] - https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/4.8.0-0.nightly-2021-04-29-071720/ [2] - https://github.com/openshift/installer/pull/4875 Best Regards.
This unblocked the builds in general, but unfortunately the resulting ppc64le binary crashes. We're going to need to revert this once we have a fix for golang to allow static linking again.
ACK Yaakov, thanks for your quick turnaround, apologies for not double checking that part.
*** Bug 1956782 has been marked as a duplicate of this bug. ***
This BZ is about Power, not ARM (which we're not building yet anyway), and I'm really not sure what Fedora 32 has to do with this. Verification steps are: 1) That the patch has landed downstream: http://pkgs.devel.redhat.com/cgit/containers/ose-installer/commit/?h=rhaos-4.8-rhel-8&id=96fcc36c437facd5b2c987ce4946c538d49ca212 http://pkgs.devel.redhat.com/cgit/containers/ose-installer-artifacts/commit/?h=rhaos-4.8-rhel-8&id=1515ad1a6c7d717e49caea217abc339a13a07be9 http://pkgs.devel.redhat.com/cgit/containers/ose-baremetal-installer/commit/?h=rhaos-4.8-rhel-8&id=151574a1384b7c8ab7cb8438bbe453d7a5628490 2) That the corresponding builds thereof have succeeded with ppc64le enabled: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600593 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600601 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600594 3) That the resulting binaries work on Power; "openshift-install version" should suffice.
[QA Summary] Here they are the corresponding logs with ppc64le enabled that corroborates the successful build after the patch has been applied: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600593 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600601 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600594 But in case of manual verification outside of native ppc64le env, this can be also checked cross-compiling with version "golang-1.16.1-3.module_el8.5.0+762+a2d12c29.x86_64", this package contains the corresponding upstream patch[1]. [Environment] ~~~ $ rpm -qa | grep golang golang-src-1.16.1-3.module_el8.5.0+762+a2d12c29.noarch golang-bin-1.16.1-3.module_el8.5.0+762+a2d12c29.x86_64 golang-1.16.1-3.module_el8.5.0+762+a2d12c29.x86_64 $ go version go version go1.16.1 linux/amd64 $ git --no-pager log --oneline --first-parent origin/master -7 db163a57d (HEAD -> master, origin/release-4.9, origin/release-4.8, origin/master, origin/HEAD) Merge pull request #4897 from kirankt/warn-delete-cluster d84a69020 Merge pull request #4905 from rna-afk/validate_publish_internal 296b0b10d Merge pull request #4924 from shiftstack/bz1940337 f7170a4e0 Merge pull request #4917 from shiftstack/BZ_1957809 c018bd4bc Merge pull request #4911 from gyohuangxin/fix-get-volume-zone f338796af Merge pull request #4908 from hardys/bmo_vendor ff06547b8 Merge pull request #4890 from multi-arch/ppc64le-link ~~~ [Results] ~~~ $ GOOS=linux GOARCH=ppc64le SKIP_GENERATION=y CC=ppc64le-linux-gnu-gcc hack/build.sh + minimum_go_version=1.14 ++ go version ++ cut -d ' ' -f 3 + current_go_version=go1.16.1 ++ version 1.16.1 ++ IFS=. ++ printf '%03d%03d%03d\n' 1 16 1 ++ unset IFS ++ version 1.14 ++ IFS=. ++ printf '%03d%03d%03d\n' 1 14 ++ unset IFS + '[' 001016001 -lt 001014000 ']' + MODE=release ++ git rev-parse --verify 'HEAD^{commit}' + GIT_COMMIT=db163a57dcfc5813de4b7e6df16685833563b98b ++ git describe --always --abbrev=40 --dirty + GIT_TAG=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b + GOFLAGS=-mod=vendor + LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b -X github.com/openshift/installer/pkg/version.Commit=db163a57dcfc5813de4b7e6df16685833563b98b' + TAGS= + OUTPUT=bin/openshift-install + export CGO_ENABLED=0 + CGO_ENABLED=0 + case "${MODE}" in + LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b -X github.com/openshift/installer/pkg/version.Commit=db163a57dcfc5813de4b7e6df16685833563b98b -s -w' + TAGS=' release' + test y '!=' y + echo ' release' + grep -q libvirt ++ go env GOARCH + test ppc64le = arm64 + go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b -X github.com/openshift/installer/pkg/version.Commit=db163a57dcfc5813de4b7e6df16685833563b98b -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install $ file bin/openshift-install bin/openshift-install: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, Go BuildID=TMdRcnmy9PF2XFVEgAH3/wgKuz-z97BRmIVUAjY3V/_2URdSKaOoC6uIl1AsvR/thUVCUOr1l8lM1R_JN3P, stripped ~~~ [1] - https://github.com/golang/go/commit/983dea90c169930e35721232afe39fd4e3fbe4a6 Best Regards.
Moving this back to ON_QA so that multi-arch can verify that it is functioning properly in our downstream builds. We appreciate Pedro's work to get this verified, but we want to prevent the risk of this getting kicked back to development if the natively compiled binaries are still problematic. Pravin - can you verify that this patch has landed in the installers you are using and is functioning as expected?
Making verification step Comment 11 un-private so that Pravin can verify
Moving this back to verified. Installation of https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.8.0-0.nightly-ppc64le-2021-05-13-211515/ was successful.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438