Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1946604

Summary: eventrouter raising "Observed a panic: &runtime.TypeAssertionError"
Product: OpenShift Container Platform Reporter: Andreas Bleischwitz <ableisch>
Component: LoggingAssignee: Vitalii Parfonov <vparfono>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.6CC: aos-bugs, gkarager, jcantril, qitang, vparfono
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: x86_64   
OS: Linux   
Whiteboard: logging-core
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-22 08:26:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Bleischwitz 2021-04-06 13:46:12 UTC 
Description of problem:
Using a pre-release image with fixes for #1859004 remains connected, but throws stack traces

Version-Release number of selected component (if applicable):
*  quay.io/vparfono/eventrouter:fixGoAway with https://github.com/openshift/eventrouter/pull/13 applied

How reproducible:
Always

Steps to Reproduce:
1. enable eventrouter

Actual results:
Occasional stack-traces are raised and eventrouter process crashes (will get restarted automatically):

Time	kubernetes.pod_name	message
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg E0327 00:52:07.946730       1 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x15579a0), concrete:(*runtime._type)(0x1600100), asserted:(*runtime._type)(0x17379e0), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg goroutine 120 [running]:
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg panic(0x1599380, 0xc0004ca3c0)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/usr/lib/golang/src/runtime/panic.go:969 +0x166
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:232
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run.func1()
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000ffdf60, 0x1982b20, 0xc00061e000, 0x1558901, 0xc00061c000)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000709760, 0x3b9aca00, 0x0, 0x1, 0xc00061c000)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg main.(*EventRouter).deleteEvent(0xc000418b80, 0x1600100, 0xc00069b600)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000709760)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:738 +0x95
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00037f4f0, 0xc00058e000)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1599380, 0xc0004ca3c0)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/eventrouter.go:197 +0xd6
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:746 +0x187
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.Until(...)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004bc500)
March 27th 2021, 01:52:07.946 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x62
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg goroutine 120 [running]:
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic(0x1599380, 0xc0004ca3c0)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/usr/lib/golang/src/runtime/panic.go:969 +0x166
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.Until(...)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x62
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/eventrouter.go:197 +0xd6
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:232
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000709760, 0x3b9aca00, 0x0, 0x1, 0xc00061c000)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event [recovered]
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Event
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg main.(*EventRouter).deleteEvent(0xc000418b80, 0x1600100, 0xc00069b600)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run.func1()
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:746 +0x187
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000709760)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000ffdf60, 0x1982b20, 0xc00061e000, 0x1558901, 0xc00061c000)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004bc500)
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg 	/go/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:738 +0x95
March 27th 2021, 01:52:07.948 eventrouter-655dd7b496-rggbg k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00037f4f0, 0xc00058e000)
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271146       1 main.go:131] Starting shared Informer(s)
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271760       1 eventrouter.go:122] Starting EvenRouter
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271072       1 interfaces.go:37] Sink is [stdout]
March 27th 2021, 01:52:09.271 eventrouter-655dd7b496-rggbg I0327 00:52:09.271176       1 main.go:118] Starting prometheus metrics.

Expected results:
No stack-traces and crashes of eventrouter.

Additional info:
Comment 1 Jeff Cantrill 2021-05-20 19:52:48 UTC 
5.1 version of this issue https://issues.redhat.com/browse/LOG-1401
Comment 2 Vitalii Parfonov 2021-05-25 12:15:56 UTC 
PR for fixes:
 - https://github.com/openshift/eventrouter/pull/19
 - https://github.com/openshift/origin-aggregated-logging/pull/2123
Comment 5 Giriyamma 2021-06-15 11:32:09 UTC 
Verified this issue using clusterlogging.4.6.0-202106021513, elasticsearch-operator.4.6.0-202106100456.
Issue is fixed, not seeing any stack-traces and crashes of eventrouter.
Comment 7 errata-xmlrpc 2021-06-22 08:26:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.35  update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2409