Bug 1370137

Summary: panic errors for unidling found in the master log
Product: OpenShift Container Platform Reporter: Meng Bo <bmeng>
Component: NodeAssignee: Solly Ross <sross>
Status: CLOSED CURRENTRELEASE QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3.0CC: aos-bugs, bbennett, eparis, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-19 15:18:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meng Bo 2016-08-25 12:01:14 UTC 
Description of problem:
There are some panic errors appear in the master log when creating pod/svc after change the enableunidling value on the nodes.

Version-Release number of selected component (if applicable):
openshift v3.3.0.25+d2ac65e-dirty
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git


How reproducible:
unknown

Steps to Reproduce:
1. Create pod/svc
# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/list_for_pods.json
2. Idle the svc and access it to unidle
3. Disable the unidling on all the nodes and restart the node service
4. Watch the master log
# journalctl -lf -u atomic-openshift-master
5. Try to create pod/svc, idle the svc and the svc to unidle


Actual results:
There are some panic errors appear in the master log sometimes.

Aug 25 19:36:57 ose-master.bmeng.local atomic-openshift-master[17442]: I0825 19:36:57.226627   17442 factory.go:212] Replication controller "u1p1/" has been deleted
Aug 25 19:36:57 ose-master.bmeng.local atomic-openshift-master[17442]: I0825 19:36:57.406172   17442 factory.go:212] Replication controller "u1p1/" has been deleted
Aug 25 19:38:31 ose-master.bmeng.local atomic-openshift-master[17442]: W0825 19:38:31.095424   17442 reflector.go:330] github.com/openshift/origin/pkg/unidling/contro
ller/controller.go:163: watch of *api.Event ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [59088/56841
]) [60087]
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: E0825 19:38:32.096053   17442 runtime.go:52] Recovered from panic: &runtime.TypeAssertionError{
interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*api.Event", missingMethod:""} (interface conversion: interface {} is
 cache.DeletedFinalStateUnknown, not *api.Event)
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:58
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:51
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:41
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/asm_amd64.s:472
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/panic.go:443
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/iface.go:182
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/pkg/unidling/controller/controller.go:118
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:183
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: <autogenerated>:27
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:253
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:127
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:103
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/
origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:88
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:89
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:49
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:103
Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/asm_amd64.s:1998


Expected results:
Should not have these errors.

Additional info:
Comment 1 Andy Goldstein 2016-08-25 13:21:05 UTC 
Solly, I think you need to do something like this: https://github.com/openshift/origin/blob/3e83a4ca2b0a10179e6cdc2ae832176532d3626c/pkg/quota/controller/clusterquotamapping/clusterquotamapping.go#L314-L330
Comment 2 Solly Ross 2016-08-25 18:02:14 UTC 
Fix here: https://github.com/openshift/origin/pull/10648
Comment 3 openshift-github-bot 2016-08-26 23:03:24 UTC 
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/bc3a9afbf2af4c6a62c52520f14b4e6e08a6584d
Unidling Controller: Deal with tombstones in cache

The delete handler in the unidling code assumed that the object passed
to the delete EventHandlerFunc would be a `*kapi.Event`.  However, it
can also be a `cache.DeletedFinalStateUnknown`, which previously caused a
panic in the controller.

Fixes bug 1370137
Comment 5 Meng Bo 2016-08-31 08:53:45 UTC 
Tested with v3.3.0.27 build, and no panic errors found.

Verify the bug.