Bug 1854115 - Installer panics when creating cluster using macOS 10.14.6
Summary: Installer panics when creating cluster using macOS 10.14.6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Mac OS
high
urgent
Target Milestone: ---
: 4.6.0
Assignee: Abhinav Dahiya
QA Contact: To Hung Sze
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-06 12:48 UTC by Dan Mace
Modified: 2020-10-27 16:12 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:12:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3885 0 None closed Bug 1854115: vendor: bump to fix tf panic on mac OS 2021-01-19 09:51:41 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:12:53 UTC

Description Dan Mace 2020-07-06 12:48:36 UTC
The latest nightly 4.6 build I pulled today[1] is causing a panic when running `create cluster` on macOS 10.14.6:

runtime: netpoll: break fd ready for -2
fatal error: runtime: netpoll: break fd ready for something unexpected

runtime stack:
runtime.throw(0xd701e49, 0x39)
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:1116 +0x72
runtime.netpoll(0x128e073de, 0x123648477b901)
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/netpoll_kqueue.go:140 +0x30a
runtime.findrunnable(0xc00008e800, 0x0)
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/proc.go:2329 +0x72b
runtime.schedule()
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/proc.go:2526 +0x2fc
runtime.park_m(0xc000340180)
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/proc.go:2696 +0x9d
runtime.mcall(0x80000)
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [runnable, locked to thread]:
reflect.Value.Pointer(0xcd7dcc0, 0x0, 0x16, 0x12)
        /opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/reflect/value.go:1429 +0x1af
github.com/golang/protobuf/proto.RegisterType(0xebdf320, 0x0, 0xd55f527, 0x12)
        /go/src/github.com/openshift/installer/vendor/github.com/golang/protobuf/proto/properties.go:486 +0x324
github.com/hashicorp/terraform-plugin-sdk/tfplugin5.init.0()
        /go/src/github.com/openshift/installer/vendor/github.com/hashicorp/terraform-plugin-sdk/tfplugin5/tfplugin5.pb.go:2651 +0x2bb

goroutine 18 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x161e40c0)
        /go/src/github.com/openshift/installer/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
        /go/src/github.com/openshift/installer/vendor/k8s.io/klog/klog.go:411 +0xd6

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00029c0a0)
        /go/src/github.com/openshift/installer/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        /go/src/github.com/openshift/installer/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57


Some related issues I found with a quick search:

https://github.com/golang/go/issues/35566
https://github.com/golang/go/issues/37726
https://go-review.googlesource.com/c/go/+/207078/
https://github.com/hashicorp/packer/issues/9032
https://github.com/hashicorp/terraform/pull/24562/commits/cd65b28da051174a13ac76e54b7bb95d3051255c
https://github.com/hashicorp/terraform/issues/24287

[1] https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest-4.6/openshift-install-mac-4.6.0-0.nightly-2020-07-06-084958.tar.gz

Comment 1 Dan Mace 2020-07-06 13:10:59 UTC
Last published working build: 4.6.0-0.nightly-2020-06-25-221502
First published broken build: 4.6.0-0.nightly-2020-06-30-020342

Comment 2 Gaoyun Pei 2020-07-08 04:20:52 UTC
Also reproduced on Mac OS X 10.13.3 with registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-07-07-233934

./openshift-install version
runtime: netpoll: break fd ready for -2
fatal error: runtime: netpoll: break fd ready for something unexpected

runtime stack:
runtime.throw(0xd914804, 0x39)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/panic.go:1116 +0x72
runtime.netpoll(0x129111c59, 0xc000091801)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/netpoll_kqueue.go:140 +0x30a
runtime.findrunnable(0xc00008f000, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/proc.go:2329 +0x72b
runtime.schedule()
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/proc.go:2526 +0x2fc
runtime.park_m(0xc000500180)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/proc.go:2696 +0x9d
runtime.mcall(0x80000)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [runnable, locked to thread]:
reflect.Value.Pointer(0xce625e0, 0x0, 0x16, 0x2b)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/reflect/value.go:1429 +0x1af
github.com/golang/protobuf/proto.RegisterType(0xee37000, 0x0, 0xd893e0a, 0x2b)
	/go/src/github.com/openshift/installer/vendor/github.com/golang/protobuf/proto/properties.go:486 +0x324
github.com/hashicorp/terraform-plugin-sdk/tfplugin5.init.0()
	/go/src/github.com/openshift/installer/vendor/github.com/hashicorp/terraform-plugin-sdk/tfplugin5/tfplugin5.pb.go:2779 +0x744

goroutine 18 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x16565fa0)
	/go/src/github.com/openshift/installer/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/openshift/installer/vendor/k8s.io/klog/klog.go:411 +0xd6

goroutine 5 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0002a0190)
	/go/src/github.com/openshift/installer/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/go/src/github.com/openshift/installer/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

Comment 3 Abhinav Dahiya 2020-07-10 17:45:47 UTC
Let's try to add the patch https://github.com/hashicorp/terraform/pull/24562/files to installer's terraform fork and see if this fixes this.

Comment 4 Dan Mace 2020-07-13 16:35:04 UTC
I built the installer with https://github.com/hashicorp/terraform/commit/cd65b28da051174a13ac76e54b7bb95d3051255c applied and verified the crash is fixed on macOS 10.14.6.

Comment 8 To Hung Sze 2020-07-16 15:43:52 UTC
I tried openshift-install-mac-4.6.0-0.nightly-2020-07-16-005008 and still see the issue.
Which build will have the fix? @adahiya

Comment 9 To Hung Sze 2020-07-17 13:19:01 UTC
Used openshift-install-mac-4.6.0-0.nightly-2020-07-16-162619 and am able to successfully install and destroy a cluster on Mac 10.13.3

Comment 10 Dan Mace 2020-07-17 18:08:47 UTC
Also verified working with openshift-client-mac-4.6.0-0.nightly-2020-07-17-043219. Thanks!

Comment 12 errata-xmlrpc 2020-10-27 16:12:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.