Bug 1530368

Summary:	atomic-openshift-master-controllers crashing due to apparent EndpointController SyncService issue
Product:	OpenShift Container Platform	Reporter:	Chris Kim <chrkim>
Component:	Master	Assignee:	Michal Fojtik <mfojtik>
Status:	CLOSED DUPLICATE	QA Contact:	Wang Haoran <haowang>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.6.0	CC:	aos-bugs, byount, erich, jokerman, mmccomas, tnozicka
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-01-03 16:06:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chris Kim 2018-01-02 19:09:21 UTC

Description of problem:

The atomic-openshift-master-controllers component of OpenShift 3.6 is crashing and causing large core dumps to be generated due to an apparent issue with the EndpointController SyncService

Version-Release number of selected component (if applicable):

3.6.x

How reproducible:

Reproducible on customer cluster, but cause unknown

Steps to Reproduce:
1. N/A

Actual results:

atomic-openshift-master-controllers crashes and generates a core dump

Expected results:

atomic-openshift-master-controllers does not crash

Additional info:

Dec 22 19:37:45 customer-master atomic-openshift-master-controllers[109251]: I1222 19:37:45.689871  109251 stateful_set.go:420] Syncing StatefulSet cust-proj1/sfs1 with 1 pods
Dec 22 19:37:45 customer-master atomic-openshift-master-controllers[109251]: I1222 19:37:45.690020  109251 stateful_set_control.go:139] StatefulSet sfs1 is waiting for Pod cache-am-0 to Terminate
Dec 22 19:37:45 customer-master atomic-openshift-master-controllers[109251]: I1222 19:37:45.690029  109251 stateful_set.go:425] Succesfully synced StatefulSet cust-proj1/sfs1 successful
Dec 22 19:37:48 customer-master atomic-openshift-master-controllers[109251]: W1222 19:37:48.422119  109251 reflector.go:323] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: watch of *v1beta1.ThirdPartyResource ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [410649612/410647538]) [410650611]
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: fatal error: concurrent map read and map write
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: goroutine 79243 [running]:
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: runtime.throw(0x516e5f2, 0x21)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /usr/lib/golang/src/runtime/panic.go:566 +0x95 fp=0xc44059f7d0 sp=0xc44059f7b0
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: runtime.mapaccess1_faststr(0x4850280, 0xc42e069c20, 0x5162114, 0x1f, 0x0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /usr/lib/golang/src/runtime/hashmap_fast.go:201 +0x4f3 fp=0xc44059f830 sp=0xc44059f7d0
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).syncService(0xc420b21dc0, 0xc447892160, 0x20, 0x0, 0x0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:461 +0x1081 fp=0xc44059fd80 sp=0xc44059f830
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).processNextWorkItem(0xc420b21dc0, 0xc428a2e600)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:365 +0x10a fp=0xc44059fe08 sp=0xc44059fd80
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).worker(0xc420b21dc0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:354 +0x2b fp=0xc44059fe28 sp=0xc44059fe08
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).(github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint.worker)-fm()
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:148 +0x2a fp=0xc44059fe40 sp=0xc44059fe28
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4322299a0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e fp=0xc44059fe78 sp=0xc44059fe40
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4322299a0, 0x3b9aca00, 0x0, 0xc42497df01, 0xc42000e1e0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad fp=0xc44059ff40 sp=0xc44059fe78
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4322299a0, 0x3b9aca00, 0xc42000e1e0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d fp=0xc44059ff78 sp=0xc44059ff40
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: runtime.goexit()
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /usr/lib/golang/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc44059ff80 sp=0xc44059ff78
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: created by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).Run
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:148 +0x189
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: goroutine 1 [select (no cases), 4014 minutes]:
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: runtime.gopark(0x0, 0x0, 0x50fe610, 0x11, 0xc4218a9910, 0x1)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /usr/lib/golang/src/runtime/proc.go:259 +0x13a fp=0xc4218f5a18 sp=0xc4218f59e8
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: runtime.block()
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /usr/lib/golang/src/runtime/select.go:224 +0x56 fp=0xc4218f5a58 sp=0xc4218f5a18
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/pkg/cmd/server/start.MasterOptions.StartMaster(0xc421044c00, 0x0, 0x0, 0x0, 0x7fffb8fd9eb3, 0x25, 0xa3ab800, 0xc42002c010, 0x0, 0x0, ...)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/start/start_master.go:215 +0x10d fp=0xc4218f5b28 sp=0xc4218f5a58
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/pkg/cmd/server/start.NewCommandStartMasterControllers.func1(0xc420453200, 0xc420e6f170, 0x0, 0x3)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/start/start_controllers.go:56 +0x146 fp=0xc4218f5ce0 sp=0xc4218f5b28
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc420453200, 0xc420e6ef90, 0x3, 0x3, 0xc420453200, 0xc420e6ef90)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:603 +0x439 fp=0xc4218f5d98 sp=0xc4218f5ce0
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4204058c0, 0xc42002c008, 0xc42002c018, 0xc4204058c0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:689 +0x367 fp=0xc4218f5ee8 sp=0xc4218f5d98
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4204058c0, 0x9, 0xc4204058c0)
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:648 +0x2b fp=0xc4218f5f18 sp=0xc4218f5ee8
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: main.main()
Dec 22 19:37:57 customer-master atomic-openshift-master-controllers[109251]: /builddir/build/BUILD/atomic-openshift-git-0.0b9377a/_output/local/go/src/github.com/openshift/origin/cmd/openshift/openshift.go:36 +0x196 fp=0xc4218f5f48 sp=0xc4218f5f18

Comment 4 Tomáš Nožička 2018-01-03 16:06:53 UTC


*** This bug has been marked as a duplicate of bug 1530367 ***