Bug 1953551 - build failure: unexpected trampoline for shared or dynamic linking
Summary: build failure: unexpected trampoline for shared or dynamic linking
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.8.0
Assignee: Yaakov Selkowitz
QA Contact: pdsilva
URL:
Whiteboard:
: 1956782 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-26 11:24 UTC by Joep van Delft
Modified: 2021-07-27 23:04 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:03:38 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4875 0 None open Bug 1953551: Link ppc64le binary dynamically 2021-04-26 20:09:10 UTC
Github openshift installer pull 4890 0 None open Bug 1953551: Revert "Link ppc64le binary dynamically" 2021-04-29 16:46:53 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:04:02 UTC

Description Joep van Delft 2021-04-26 11:24:38 UTC
Version:

$ openshift-install version
4.8

Platform:

#Please specify the platform type: aws, libvirt, openstack or baremetal etc.

Please specify:
* IPI (automated install with `openshift-install`. If you don't know, then it's IPI)
* UPI (semi-manual installation on customized infrastructure)

What happened?
4.8 builds of ose-installer are failing for ppc64le: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36401931

error:
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=v4.8.0 -X github.com/openshift/installer/pkg/version.Commit=18974847b6a68fef21f3fae7320259f4ed3beff8 -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
# github.com/openshift/installer/cmd/openshift-install
github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostService).Service: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingRootPasswordRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingRootPasswordRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsServiceAddUsingSshRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsService).HostService: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsService).HostService: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*HostsService).String: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking
github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking
/usr/lib/golang/pkg/tool/linux_ppc64le/link: too many errors
running 'hack/build.sh' failed with exit code 2


What did you expect to happen?
A working build

How to reproduce it (as minimally and precisely as possible)?

$ your-commands-here

Anything else we need to know?

#Enter text here.

Comment 1 Yaakov Selkowitz 2021-04-26 18:22:31 UTC
This looks related: https://github.com/golang/go/issues/45564

Comment 5 Pedro Amoedo 2021-04-29 14:22:59 UTC
[QA Summary]

I can see that the version "4.8.0-0.nightly-2021-04-29-071720" was "Accepted" 7 hours ago and also published as dev-preview here[1], this version contains the expected PR[2], and therefore also the pending ones since previous commit "fdf0269c3":

~~~
$ oc adm release info --commits registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-29-071720 --registry-config=pull-secret-ci | grep ' installer '
  installer                                      https://github.com/openshift/installer                                      52fef9e1d7b765470e493670177eefd73bc27c6f

$ git show 52fef9e1d7b765470e493670177eefd73bc27c6f
commit 52fef9e1d7b765470e493670177eefd73bc27c6f
Merge: 911eb7f69 24ac0a15e
Author: OpenShift Merge Robot <openshift-merge-robot@users.noreply.github.com>
Date:   Wed Apr 28 18:55:07 2021 -0400

    Merge pull request #4875 from multi-arch/ppc64le-link
    
    Bug 1953551: Link ppc64le binary dynamically

$ git --no-pager log --oneline --first-parent 52fef9e1d -3
52fef9e1d Merge pull request #4875 from multi-arch/ppc64le-link
911eb7f69 Merge pull request #4868 from wking/stable-4.8
18974847b Merge pull request #4755 from staebler/user_tags_in_cluster
~~~

[1] - https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/4.8.0-0.nightly-2021-04-29-071720/
[2] - https://github.com/openshift/installer/pull/4875

Best Regards.

Comment 6 Yaakov Selkowitz 2021-04-29 14:54:33 UTC
This unblocked the builds in general, but unfortunately the resulting ppc64le binary crashes.  We're going to need to revert this once we have a fix for golang to allow static linking again.

Comment 7 Pedro Amoedo 2021-04-29 15:59:03 UTC
ACK Yaakov, thanks for your quick turnaround, apologies for not double checking that part.

Comment 8 Yaakov Selkowitz 2021-05-04 13:37:05 UTC
*** Bug 1956782 has been marked as a duplicate of this bug. ***

Comment 21 Pedro Amoedo 2021-05-12 08:32:33 UTC
[QA Summary]

Here they are the corresponding logs with ppc64le enabled that corroborates the successful build after the patch has been applied:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600593
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600601
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1600594

But in case of manual verification outside of native ppc64le env, this can be also checked cross-compiling with version "golang-1.16.1-3.module_el8.5.0+762+a2d12c29.x86_64", this package contains the corresponding upstream patch[1].

[Environment]

~~~
$ rpm -qa | grep golang
golang-src-1.16.1-3.module_el8.5.0+762+a2d12c29.noarch
golang-bin-1.16.1-3.module_el8.5.0+762+a2d12c29.x86_64
golang-1.16.1-3.module_el8.5.0+762+a2d12c29.x86_64

$ go version
go version go1.16.1 linux/amd64

$ git --no-pager log --oneline --first-parent origin/master -7
db163a57d (HEAD -> master, origin/release-4.9, origin/release-4.8, origin/master, origin/HEAD) Merge pull request #4897 from kirankt/warn-delete-cluster
d84a69020 Merge pull request #4905 from rna-afk/validate_publish_internal
296b0b10d Merge pull request #4924 from shiftstack/bz1940337
f7170a4e0 Merge pull request #4917 from shiftstack/BZ_1957809
c018bd4bc Merge pull request #4911 from gyohuangxin/fix-get-volume-zone
f338796af Merge pull request #4908 from hardys/bmo_vendor
ff06547b8 Merge pull request #4890 from multi-arch/ppc64le-link
~~~

[Results]

~~~
$ GOOS=linux GOARCH=ppc64le SKIP_GENERATION=y CC=ppc64le-linux-gnu-gcc hack/build.sh
+ minimum_go_version=1.14
++ go version
++ cut -d ' ' -f 3
+ current_go_version=go1.16.1
++ version 1.16.1
++ IFS=.
++ printf '%03d%03d%03d\n' 1 16 1
++ unset IFS
++ version 1.14
++ IFS=.
++ printf '%03d%03d%03d\n' 1 14
++ unset IFS
+ '[' 001016001 -lt 001014000 ']'
+ MODE=release
++ git rev-parse --verify 'HEAD^{commit}'
+ GIT_COMMIT=db163a57dcfc5813de4b7e6df16685833563b98b
++ git describe --always --abbrev=40 --dirty
+ GIT_TAG=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b
+ GOFLAGS=-mod=vendor
+ LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b -X github.com/openshift/installer/pkg/version.Commit=db163a57dcfc5813de4b7e6df16685833563b98b'
+ TAGS=
+ OUTPUT=bin/openshift-install
+ export CGO_ENABLED=0
+ CGO_ENABLED=0
+ case "${MODE}" in
+ LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b -X github.com/openshift/installer/pkg/version.Commit=db163a57dcfc5813de4b7e6df16685833563b98b -s -w'
+ TAGS=' release'
+ test y '!=' y
+ echo ' release'
+ grep -q libvirt
++ go env GOARCH
+ test ppc64le = arm64
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4580-gdb163a57dcfc5813de4b7e6df16685833563b98b -X github.com/openshift/installer/pkg/version.Commit=db163a57dcfc5813de4b7e6df16685833563b98b -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install

$ file bin/openshift-install 
bin/openshift-install: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, Go BuildID=TMdRcnmy9PF2XFVEgAH3/wgKuz-z97BRmIVUAjY3V/_2URdSKaOoC6uIl1AsvR/thUVCUOr1l8lM1R_JN3P, stripped
~~~

[1] - https://github.com/golang/go/commit/983dea90c169930e35721232afe39fd4e3fbe4a6

Best Regards.

Comment 22 Jeremy Poulin 2021-05-12 19:19:42 UTC
Moving this back to ON_QA so that multi-arch can verify that it is functioning properly in our downstream builds. We appreciate Pedro's work to get this verified, but we want to prevent the risk of this getting kicked back to development if the natively compiled binaries are still problematic.

Pravin - can you verify that this patch has landed in the installers you are using and is functioning as expected?

Comment 23 Dan Li 2021-05-13 11:40:20 UTC
Making verification step Comment 11 un-private so that Pravin can verify

Comment 25 Jeremy Poulin 2021-05-15 00:53:13 UTC
Moving this back to verified.
Installation of https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.8.0-0.nightly-ppc64le-2021-05-13-211515/ was successful.

Comment 29 errata-xmlrpc 2021-07-27 23:03:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.