Bug 1956782

Summary:	OCP 4.8 latest build fails to deploy at step: openshift-install create manifests
Product:	OpenShift Container Platform	Reporter:	Julie <mjulie>
Component:	Installer	Assignee:	aos-install
Installer sub component:	openshift-installer	QA Contact:	Gaoyun Pei <gpei>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	aos-bugs, danili, mhamzy, yselkowi
Version:	4.8	Flags:	mjulie: needinfo-
Target Milestone:	---
Target Release:	---
Hardware:	ppc64le
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-04 13:37:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Julie 2021-05-04 12:04:47 UTC

Description of problem:
OCP 4.8 installlation is failing with latest nightly build 


Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Use the latest nightly build -> https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.8.0-0.nightly-ppc64le-2021-05-04-055625/

2. Try deploying on Power (ppc64le) platform 


Actual results:
OCP 4.8 deploy fails at this step

# openshift-install create manifests --log-level debug

unexpected fault address 0xffffffffffff42ca
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0xffffffffffff42ca pc=0x1a6abff4]
goroutine 1 [running, locked to thread]:
runtime.throw(0x217384d8, 0x5)
        /usr/lib/golang/src/runtime/panic.go:1117 +0x5c fp=0xc0009674f8 sp=0xc0009674b8 pc=0x152dfc3c
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:741 +0x274 fp=0xc000967540 sp=0xc0009674f8 pc=0x152f78f4
github.com/aws/aws-sdk-go/service/efs.init()
        /go/src/github.com/openshift/installer/vendor/github.com/aws/aws-sdk-go/service/efs/errors.go:194 +0x24 fp=0xc0009675b0 sp=0xc000967560 pc=0x18ac4934
runtime.doInit(0x297b8740)
        /usr/lib/golang/src/runtime/proc.go:6265 +0xf0 fp=0xc000967710 sp=0xc0009675b0 pc=0x152f0910
runtime.doInit(0x298099a0)
        /usr/lib/golang/src/runtime/proc.go:6242 +0x64 fp=0xc000967870 sp=0xc000967710 pc=0x152f0884
runtime.doInit(0x297d7200)
        /usr/lib/golang/src/runtime/proc.go:6242 +0x64 fp=0xc0009679d0 sp=0xc000967870 pc=0x152f0884
runtime.doInit(0x297ca8e0)
        /usr/lib/golang/src/runtime/proc.go:6242 +0x64 fp=0xc000967b30 sp=0xc0009679d0 pc=0x152f0884
runtime.doInit(0x297e4040)
        /usr/lib/golang/src/runtime/proc.go:6242 +0x64 fp=0xc000967c90 sp=0xc000967b30 pc=0x152f0884
runtime.doInit(0x297c0bc0)
        /usr/lib/golang/src/runtime/proc.go:6242 +0x64 fp=0xc000967df0 sp=0xc000967c90 pc=0x152f0884
runtime.doInit(0x297e6e60)
        /usr/lib/golang/src/runtime/proc.go:6242 +0x64 fp=0xc000967f50 sp=0xc000967df0 pc=0x152f0884
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:208 +0x214 fp=0xc000967fc0 sp=0xc000967f50 pc=0x152e2344
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_ppc64x.s:869 +0x4 fp=0xc000967fc0 sp=0xc000967fc0 pc=0x15315b24
goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x29970820)
        /go/src/github.com/openshift/installer/vendor/k8s.io/klog/v2/klog.go:1164 +0x78
created by k8s.io/klog/v2.init.0
        /go/src/github.com/openshift/installer/vendor/k8s.io/klog/v2/klog.go:418 +0xe4
goroutine 22 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000518500)
        /go/src/github.com/openshift/installer/vendor/go.opencensus.io/stats/view/worker.go:276 +0xb4
created by go.opencensus.io/stats/view.init.0
        /go/src/github.com/openshift/installer/vendor/go.opencensus.io/stats/view/worker.go:34 +0x78
goroutine 6 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x29970740)
        /go/src/github.com/openshift/installer/vendor/k8s.io/klog/klog.go:1010 +0x78
created by k8s.io/klog.init.0
        /go/src/github.com/openshift/installer/vendor/k8s.io/klog/klog.go:411 +0xe0
goroutine 35 [sleep]:
time.Sleep(0x6fc23ac00)
        /usr/lib/golang/src/runtime/time.go:193 +0xd8
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
        /go/src/github.com/openshift/installer/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:63 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
        /go/src/github.com/openshift/installer/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:62 +0x34


Expected results:
OCP 4.8 deploy should complete sucessfully.


Additional info:

Comment 1 Adam Kaplan 2021-05-04 13:17:47 UTC

The "Build" component is for issues related to OpenShift's in cluster build capabilities. Moving this issue to the installer team.

Is openshift-installer running on a Power machine, or in a different environment?

Comment 2 Mark Hamzy 2021-05-04 13:19:22 UTC

There are a number of go issues opened which need to be in the build root:

cmd/link: unexpected trampoline when cross-compiling to ppc64le
https://github.com/golang/go/issues/45564

cmd/link: SIGSEGV running 'openshift-install version' for release-4.8 using external linking on PPC64LE
https://github.com/golang/go/issues/45850

Comment 3 Mark Hamzy 2021-05-04 13:21:23 UTC

This also needs to be pulled into the installer:

Bug 1953551: Revert "Link ppc64le binary dynamically"
https://github.com/openshift/installer/pull/4890

Comment 4 Mark Hamzy 2021-05-04 13:23:49 UTC

This is also a potential workaround:

https://github.com/golang/go/issues/45850#issuecomment-829459432

Comment 5 Dan Li 2021-05-04 13:34:16 UTC

Hi Installer team/assignee team, the multi-arch team evaluated the bug and we think that this bug should be a "Blocker+" due to its impact.

Comment 6 Yaakov Selkowitz 2021-05-04 13:37:05 UTC


*** This bug has been marked as a duplicate of bug 1953551 ***

Comment 7 Julie 2021-05-10 05:54:14 UTC

(In reply to Adam Kaplan from comment #1)
> The "Build" component is for issues related to OpenShift's in cluster build
> capabilities. Moving this issue to the installer team.
> 
> Is openshift-installer running on a Power machine, or in a different
> environment?

yes, running on Power (ppc64le arch).