Bug 1701495
| Summary: | Eventrouter pod in CrashLoopBackOff status: "runtime error: invalid memory address or nil pointer dereference" | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> | |
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | |
| Status: | CLOSED ERRATA | QA Contact: | Qiaoling Tang <qitang> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.1.0 | CC: | aos-bugs, clasohm, david, jcantril, jmalde, nhosoi, rmeggins, skrenger, sreber, vlaad | |
| Target Milestone: | --- | |||
| Target Release: | 4.3.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openshift-ose-logging-eventrouter:v4.3.0-201910250623 | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Eventrouter was not able to handle events of type "Info" or other event types unknown to it.
Consequence: Eventrouter would crash trying to handle unknown event types.
Fix: Add explicit support for "Info" types, and otherwise fix handling for unknown event types.
Result: Eventrouter properly handles "Info" and unknown event types.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1762409 (view as bug list) | Environment: | ||
| Last Closed: | 2020-01-23 11:03:45 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1762409, 1764312, 1764314 | |||
|
Description
Qiaoling Tang
2019-04-19 09:21:02 UTC
This issue also could be reproduced in env installed by IPI in AWS. I was able to reproduce this issue by manually creating an event with `"type": "Info"`:
1. Set up a cluster (I used v3.11.141), with `openshift3/ose-logging-eventrouter:v3.11.141-2`
2. Run `oc apply -f event.json` with any new event with `"type": "Info"`:
```
{
"apiVersion": "v1",
"count": 1,
"eventTime": null,
"involvedObject": {
"apiVersion": "apps.openshift.io/v1",
"kind": "DeploymentConfig",
"name": "router",
"namespace": "default",
},
"kind": "Event",
"message": "My test",
"metadata": {
"name": "test-event",
"namespace": "default"
},
"reason": "DeploymentCreated",
"reportingComponent": "",
"reportingInstance": "",
"source": {
"component": "deploymentconfig-controller"
},
"type": "Info"
}
```
3. Observe that `eventrouter` crashes immediately and stays in `CrashLoopBackOff`:
```
$ oc get po
NAME READY STATUS RESTARTS AGE
[..]
logging-eventrouter-1-gv6jp 0/1 CrashLoopBackOff 1 6m
[..]
```
4. Observe the error message:
```
$ oc logs logging-eventrouter-1-gv6jp
[..]
E0912 14:23:42.370996 1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/lib/golang/src/runtime/asm_amd64.s:514
/usr/lib/golang/src/runtime/panic.go:489
/usr/lib/golang/src/runtime/panic.go:63
/usr/lib/golang/src/runtime/signal_unix.go:290
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/eventrouter.go:152
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/eventrouter.go:113
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/eventrouter.go:86
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:192
<autogenerated>:54
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547
/usr/lib/golang/src/runtime/asm_amd64.s:2197
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x11c2239]
goroutine 35 [running]:
github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x130ff20, 0x1d2c5b0)
/usr/lib/golang/src/runtime/panic.go:489 +0x2cf
main.prometheusEvent(0xc4205ecd38)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/eventrouter.go:152 +0xe9
main.(*EventRouter).addEvent(0xc42012d340, 0x1471800, 0xc4205ecd38)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/eventrouter.go:113 +0x3c
main.(*EventRouter).(main.addEvent)-fm(0x1471800, 0xc4205ecd38)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/eventrouter.go:86 +0x3e
github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(0xc4201a74a0, 0xc4201a74b0, 0xc4201a74c0, 0x1471800, 0xc4205ecd38)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/controller.go:192 +0x49
github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd(0xc42000ca60, 0x1471800, 0xc4205ecd38)
<autogenerated>:54 +0x73
github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc42009f6b0, 0xc420418120)
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547 +0x2f6
created by github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run
/builddir/build/BUILD/eventrouter-5bd925143c4263d18cb33bfc3f3edbcf4267b2a5/src/github.com/openshift/eventrouter/vendor/k8s.io/client-go/tools/cache/shared_informer.go:402 +0xb3
```
Not a blocker for 4.2. Moving to 4.3 and likely will need to be backported to 4.2 and prior Verified with ose-logging-eventrouter-v4.3.0-201910250623. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 |