Bug 1370137 - panic errors for unidling found in the master log
Summary: panic errors for unidling found in the master log
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Solly Ross
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-25 12:01 UTC by Meng Bo
Modified: 2016-09-19 15:18 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-19 15:18:45 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC
Origin (Github) 10648 None None None 2016-08-25 21:42:24 UTC

Description Meng Bo 2016-08-25 12:01:14 UTC
Description of problem:
There are some panic errors appear in the master log when creating pod/svc after change the enableunidling value on the nodes.

Version-Release number of selected component (if applicable):
openshift v3.3.0.25+d2ac65e-dirty
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git


How reproducible:
unknown

Steps to Reproduce:
1. Create pod/svc
# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/list_for_pods.json
2. Idle the svc and access it to unidle
3. Disable the unidling on all the nodes and restart the node service
4. Watch the master log
# journalctl -lf -u atomic-openshift-master
5. Try to create pod/svc, idle the svc and the svc to unidle


Actual results:
There are some panic errors appear in the master log sometimes.

Aug 25 19:36:57 ose-master.bmeng.local atomic-openshift-master[17442]: I0825 19:36:57.226627   17442 factory.go:212] Replication controller "u1p1/" has been deleted
Aug 25 19:36:57 ose-master.bmeng.local atomic-openshift-master[17442]: I0825 19:36:57.406172   17442 factory.go:212] Replication controller "u1p1/" has been deleted
Aug 25 19:38:31 ose-master.bmeng.local atomic-openshift-master[17442]: W0825 19:38:31.095424   17442 reflector.go:330] github.com/openshift/origin/pkg/unidling/contro
ller/controller.go:163: watch of *api.Event ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [59088/56841
]) [60087]
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: E0825 19:38:32.096053   17442 runtime.go:52] Recovered from panic: &runtime.TypeAssertionError{
interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*api.Event", missingMethod:""} (interface conversion: interface {} is
 cache.DeletedFinalStateUnknown, not *api.Event)
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:58
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:51
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:41
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/asm_amd64.s:472
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/panic.go:443
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/iface.go:182
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/pkg/unidling/controller/controller.go:118
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:183
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: <autogenerated>:27
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:253
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:127
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:103
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:88
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:89
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:49
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:103
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/asm_amd64.s:1998


Expected results:
Should not have these errors.

Additional info:

Comment 2 Solly Ross 2016-08-25 18:02:14 UTC
Fix here: https://github.com/openshift/origin/pull/10648

Comment 3 openshift-github-bot 2016-08-26 23:03:24 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/bc3a9afbf2af4c6a62c52520f14b4e6e08a6584d
Unidling Controller: Deal with tombstones in cache

The delete handler in the unidling code assumed that the object passed
to the delete EventHandlerFunc would be a `*kapi.Event`.  However, it
can also be a `cache.DeletedFinalStateUnknown`, which previously caused a
panic in the controller.

Fixes bug 1370137

Comment 5 Meng Bo 2016-08-31 08:53:45 UTC
Tested with v3.3.0.27 build, and no panic errors found.

Verify the bug.


Note You need to log in before you can comment on or make changes to this bug.