Red Hat Bugzilla – Bug 1370137
panic errors for unidling found in the master log
Last modified: 2016-09-19 11:18:45 EDT
Description of problem: There are some panic errors appear in the master log when creating pod/svc after change the enableunidling value on the nodes. Version-Release number of selected component (if applicable): openshift v3.3.0.25+d2ac65e-dirty kubernetes v1.3.0+507d3a7 etcd 2.3.0+git How reproducible: unknown Steps to Reproduce: 1. Create pod/svc # oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/list_for_pods.json 2. Idle the svc and access it to unidle 3. Disable the unidling on all the nodes and restart the node service 4. Watch the master log # journalctl -lf -u atomic-openshift-master 5. Try to create pod/svc, idle the svc and the svc to unidle Actual results: There are some panic errors appear in the master log sometimes. Aug 25 19:36:57 ose-master.bmeng.local atomic-openshift-master[17442]: I0825 19:36:57.226627 17442 factory.go:212] Replication controller "u1p1/" has been deleted Aug 25 19:36:57 ose-master.bmeng.local atomic-openshift-master[17442]: I0825 19:36:57.406172 17442 factory.go:212] Replication controller "u1p1/" has been deleted Aug 25 19:38:31 ose-master.bmeng.local atomic-openshift-master[17442]: W0825 19:38:31.095424 17442 reflector.go:330] github.com/openshift/origin/pkg/unidling/contro ller/controller.go:163: watch of *api.Event ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [59088/56841 ]) [60087] Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: E0825 19:38:32.096053 17442 runtime.go:52] Recovered from panic: &runtime.TypeAssertionError{ interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*api.Event", missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *api.Event) Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:58 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:51 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:41 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/asm_amd64.s:472 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/panic.go:443 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/iface.go:182 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/pkg/unidling/controller/controller.go:118 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:183 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: <autogenerated>:27 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:253 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:127 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/ origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:88 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:89 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:49 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 Aug 25 19:38:32 ose-master.bmeng.local atomic-openshift-master[17442]: /usr/lib/golang/src/runtime/asm_amd64.s:1998 Expected results: Should not have these errors. Additional info:
Solly, I think you need to do something like this: https://github.com/openshift/origin/blob/3e83a4ca2b0a10179e6cdc2ae832176532d3626c/pkg/quota/controller/clusterquotamapping/clusterquotamapping.go#L314-L330
Fix here: https://github.com/openshift/origin/pull/10648
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/bc3a9afbf2af4c6a62c52520f14b4e6e08a6584d Unidling Controller: Deal with tombstones in cache The delete handler in the unidling code assumed that the object passed to the delete EventHandlerFunc would be a `*kapi.Event`. However, it can also be a `cache.DeletedFinalStateUnknown`, which previously caused a panic in the controller. Fixes bug 1370137
Tested with v3.3.0.27 build, and no panic errors found. Verify the bug.