Bug 1530367

Summary: atomic-openshift-master-controllers crashing due to apparent StatefulSet isRunningAndReady issue
Product: OpenShift Container Platform Reporter: Chris Kim <chrkim>
Component: MasterAssignee: Tomáš Nožička <tnozicka>
Status: CLOSED ERRATA QA Contact: Wang Haoran <haowang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6.0CC: aos-bugs, byount, chrkim, dmace, erich, jokerman, mfojtik, mmccomas
Target Milestone: ---   
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-12 05:59:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Kim 2018-01-02 19:02:44 UTC 
Description of problem:

The atomic-openshift-master-controllers component of OpenShift 3.6 is crashing and causing large core dumps to be generated due to an apparent issue with StatefulSet's + Init Container Annotations + Checking for Running/Ready State

Version-Release number of selected component (if applicable):

3.6.x

How reproducible:

Reproducible on customer cluster

Steps to Reproduce:
1. Create stateful sets with init container annotations

Actual results:

atomic-openshift-master-controllers panics and creates core dump

Expected results:

atomic-openshift-master-controllers doesn't panic and no core dump is created

Additional info:

Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694188   32109 stateful_set.go:425] Succesfully synced StatefulSet customer-sfs-proj1/sfs1 successful
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694244   32109 stateful_set.go:420] Syncing StatefulSet customer-sfs-proj2/sfs2 with 1 pods
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694259   32109 stateful_set.go:425] Succesfully synced StatefulSet customer-sfs-proj2/sfs2 successful
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694283   32109 stateful_set.go:420] Syncing StatefulSet customer-sfs-proj3/sfs3 with 1 pods
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694297   32109 stateful_set.go:425] Succesfully synced StatefulSet customer-sfs-proj3/sfs3 successful
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694322   32109 stateful_set.go:420] Syncing StatefulSet customer-sfs-proj4/sfs4 with 1 pods
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: fatal error: concurrent map read and map write
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: goroutine 101683 [running]:
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: runtime.throw(0x516e5f2, 0x21)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /usr/lib/golang/src/runtime/panic.go:566 +0x95 fp=0xc45eb99868 sp=0xc45eb99848
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: runtime.mapaccess2_faststr(0x4850280, 0xc45c8eaea0, 0x517db6a, 0x23, 0xc42f7d1001, 0x0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /usr/lib/golang/src/runtime/hashmap_fast.go:306 +0x52b fp=0xc45eb998c8 sp=0xc45eb99868
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.isRunningAndReady(0xc459538000, 0xc42f7d1020)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set_utils.go:200 +0xec fp=0xc45eb999a8 sp=0xc45eb998c8
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*defaultStatefulSetControl).UpdateStatefulSet(0xc4234ce9a0, 0xc447aba380, 0xc45d4a6778, 0x1, 0x1, 0x3, 0x21735b0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set_control.go:145 +0x683 fp=0xc45eb99b68 sp=0xc45eb999a8
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).syncStatefulSet(0xc427a60c00, 0xc447aba380, 0xc45d4a6778, 0x1, 0x1, 0x1, 0x1)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:421 +0x1d3 fp=0xc45eb99c60 sp=0xc45eb99b68
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).sync(0xc427a60c00, 0xc42582ade0, 0x23, 0x0, 0x0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:415 +0x392 fp=0xc45eb99d60 sp=0xc45eb99c60
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).processNextWorkItem(0xc427a60c00, 0xc42791ce00)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:367 +0x125 fp=0xc45eb99e08 sp=0xc45eb99d60
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).worker(0xc427a60c00)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:378 +0x2b fp=0xc45eb99e28 sp=0xc45eb99e08
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).(github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.worker)-fm()
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:152 +0x2a fp=0xc45eb99e40 sp=0xc45eb99e28
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc425a384d0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e fp=0xc45eb99e78 sp=0xc45eb99e40
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc425a384d0, 0x3b9aca00, 0x0, 0x1, 0xc42000e1e0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad fp=0xc45eb99f40 sp=0xc45eb99e78
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc425a384d0, 0x3b9aca00, 0xc42000e1e0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d fp=0xc45eb99f78 sp=0xc45eb99f40
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: runtime.goexit()
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /usr/lib/golang/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc45eb99f80 sp=0xc45eb99f78
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: created by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).Run
Comment 1 Eric Rich 2018-01-02 21:12:34 UTC 
I wonder if disabling the TP api would mitigate this?
 
Following: https://kubernetes.io/docs/concepts/overview/kubernetes-api/

> extensions/v1/beta1/statefulsets=false
Comment 10 Tomáš Nožička 2018-01-03 16:06:53 UTC 
*** Bug 1530368 has been marked as a duplicate of this bug. ***
Comment 13 Wang Haoran 2018-01-17 05:29:35 UTC 
Moved to Verified per last comment.
Comment 16 errata-xmlrpc 2018-04-12 05:59:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1106