Bug 1762409

Summary: Eventrouter pod in CrashLoopBackOff status: "runtime error: invalid memory address or nil pointer dereference"
Product: OpenShift Container Platform Reporter: Rich Megginson <rmeggins>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Qiaoling Tang <qitang>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.0CC: anli, aos-bugs, clasohm, david, jcantril, jmalde, nhosoi, qitang, rmeggins, skrenger, sreber
Target Milestone: ---   
Target Release: 4.2.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ose-logging-eventrouter:v4.2.2-201910250432 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1701495
: 1764312 (view as bug list) Environment:
Last Closed: 2019-11-13 18:55:51 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1701495    
Bug Blocks: 1764312, 1764314    

Comment 1 Anping Li 2019-10-21 09:12:00 UTC 
Hit this issue using openshift/ose-logging-eventrouter:v4.2.1-201910171309 after 81m.

cluster-logging-eventrouter-568995ddb6-gjtg5    0/1     CrashLoopBackOff   20         81m

E1021 09:07:37.241985       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:522
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:513
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:82
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/signal_unix.go:390
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/eventrouter.go:152
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/eventrouter.go:113
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/eventrouter.go:86
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:195
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381
/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xf556cc]

goroutine 7 [running]:
github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x1077880, 0x1d28f20)
	/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:513 +0x1b9
main.prometheusEvent(0xc001845400)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/eventrouter.go:152 +0xdc
main.(*EventRouter).addEvent(0xc0003e4040, 0x11b3900, 0xc001845400)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/eventrouter.go:113 +0x3c
main.(*EventRouter).addEvent-fm(0x11b3900, 0xc001845400)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/eventrouter.go:86 +0x3e
github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(0xc0004d8020, 0xc0004d8030, 0xc0004d8040, 0x11b3900, 0xc001845400)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49
github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000b00a0)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 +0x24a
github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm()
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x2a
github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00013e138, 0xc0001e8000)
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/builddir/build/BUILD/eventrouter-ed73fb6584b98a08390db56f1337a45c57044df1/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62
Comment 5 Qiaoling Tang 2019-10-30 05:21:56 UTC 
Verified with ose-logging-eventrouter-v4.2.3-201910291027.
Comment 6 Rich Megginson 2019-10-30 13:38:11 UTC 
Moving this back to POST so that the bugzilla bot will merge https://github.com/openshift/origin-aggregated-logging/pull/1768
Comment 9 Qiaoling Tang 2019-11-06 05:46:54 UTC 
Verified with ose-logging-eventrouter-v4.2.4-201911050122
Comment 11 errata-xmlrpc 2019-11-13 18:55:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3303