Bug 1530367 - atomic-openshift-master-controllers crashing due to apparent StatefulSet isRunningAndReady issue
Summary: atomic-openshift-master-controllers crashing due to apparent StatefulSet isRu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.6.z
Assignee: Tomáš Nožička
QA Contact: Wang Haoran
URL:
Whiteboard:
: 1530368 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-02 19:02 UTC by Chris Kim
Modified: 2021-03-11 16:47 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-04-12 05:59:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1106 0 None None None 2018-04-12 06:00:45 UTC

Description Chris Kim 2018-01-02 19:02:44 UTC 
Description of problem:

The atomic-openshift-master-controllers component of OpenShift 3.6 is crashing and causing large core dumps to be generated due to an apparent issue with StatefulSet's + Init Container Annotations + Checking for Running/Ready State

Version-Release number of selected component (if applicable):

3.6.x

How reproducible:

Reproducible on customer cluster

Steps to Reproduce:
1. Create stateful sets with init container annotations

Actual results:

atomic-openshift-master-controllers panics and creates core dump

Expected results:

atomic-openshift-master-controllers doesn't panic and no core dump is created

Additional info:

Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694188   32109 stateful_set.go:425] Succesfully synced StatefulSet customer-sfs-proj1/sfs1 successful
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694244   32109 stateful_set.go:420] Syncing StatefulSet customer-sfs-proj2/sfs2 with 1 pods
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694259   32109 stateful_set.go:425] Succesfully synced StatefulSet customer-sfs-proj2/sfs2 successful
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694283   32109 stateful_set.go:420] Syncing StatefulSet customer-sfs-proj3/sfs3 with 1 pods
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694297   32109 stateful_set.go:425] Succesfully synced StatefulSet customer-sfs-proj3/sfs3 successful
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: I1220 00:42:49.694322   32109 stateful_set.go:420] Syncing StatefulSet customer-sfs-proj4/sfs4 with 1 pods
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: fatal error: concurrent map read and map write
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: goroutine 101683 [running]:
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: runtime.throw(0x516e5f2, 0x21)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /usr/lib/golang/src/runtime/panic.go:566 +0x95 fp=0xc45eb99868 sp=0xc45eb99848
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: runtime.mapaccess2_faststr(0x4850280, 0xc45c8eaea0, 0x517db6a, 0x23, 0xc42f7d1001, 0x0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /usr/lib/golang/src/runtime/hashmap_fast.go:306 +0x52b fp=0xc45eb998c8 sp=0xc45eb99868
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.isRunningAndReady(0xc459538000, 0xc42f7d1020)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set_utils.go:200 +0xec fp=0xc45eb999a8 sp=0xc45eb998c8
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*defaultStatefulSetControl).UpdateStatefulSet(0xc4234ce9a0, 0xc447aba380, 0xc45d4a6778, 0x1, 0x1, 0x3, 0x21735b0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set_control.go:145 +0x683 fp=0xc45eb99b68 sp=0xc45eb999a8
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).syncStatefulSet(0xc427a60c00, 0xc447aba380, 0xc45d4a6778, 0x1, 0x1, 0x1, 0x1)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:421 +0x1d3 fp=0xc45eb99c60 sp=0xc45eb99b68
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).sync(0xc427a60c00, 0xc42582ade0, 0x23, 0x0, 0x0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:415 +0x392 fp=0xc45eb99d60 sp=0xc45eb99c60
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).processNextWorkItem(0xc427a60c00, 0xc42791ce00)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:367 +0x125 fp=0xc45eb99e08 sp=0xc45eb99d60
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).worker(0xc427a60c00)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:378 +0x2b fp=0xc45eb99e28 sp=0xc45eb99e08
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).(github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.worker)-fm()
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset/stateful_set.go:152 +0x2a fp=0xc45eb99e40 sp=0xc45eb99e28
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc425a384d0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e fp=0xc45eb99e78 sp=0xc45eb99e40
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc425a384d0, 0x3b9aca00, 0x0, 0x1, 0xc42000e1e0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad fp=0xc45eb99f40 sp=0xc45eb99e78
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc425a384d0, 0x3b9aca00, 0xc42000e1e0)
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d fp=0xc45eb99f78 sp=0xc45eb99f40
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: runtime.goexit()
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: /usr/lib/golang/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc45eb99f80 sp=0xc45eb99f78
Dec 20 00:42:49 customer-master atomic-openshift-master-controllers[32109]: created by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/statefulset.(*StatefulSetController).Run
Comment 1 Eric Rich 2018-01-02 21:12:34 UTC 
I wonder if disabling the TP api would mitigate this?
 
Following: https://kubernetes.io/docs/concepts/overview/kubernetes-api/

> extensions/v1/beta1/statefulsets=false
Comment 10 Tomáš Nožička 2018-01-03 16:06:53 UTC 
*** Bug 1530368 has been marked as a duplicate of this bug. ***
Comment 13 Wang Haoran 2018-01-17 05:29:35 UTC 
Moved to Verified per last comment.
Comment 16 errata-xmlrpc 2018-04-12 05:59:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1106
Note You need to log in before you can comment on or make changes to this bug.