Description of problem: During a OCP 3.7 reliability test long run, 8 days into the test, the panic below was seen. Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: E1106 09:04:09.512095 36356 runtime.go:66] Observed a panic: &runtime.TypeAssertionError{interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*apps.DeploymentConfig", missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *apps.DeploymentConfig) Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/asm_amd64.s:514 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/panic.go:489 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/iface.go:172 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/pkg/image/trigger/deploymentconfigs/deploymentconfigs.go:118 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: <autogenerated>:1 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/pkg/image/controller/trigger/cache.go:81 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:206 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: <autogenerated>:59 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /builddir/build/BUILD/atomic-openshift-git-0.d787960/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:549 Nov 06 09:04:09 ip-172-31-29-26.us-west-2.compute.internal atomic-openshift-master-controllers[36345]: /usr/lib/golang/src/runtime/asm_amd64.s:2197 The reliability test is a 4 node cluster (1 master, 1 infra, 2 compute) which performs the following actions on scheduled basis at "normal" intervals. It is not a stress test. The purpose of the test is to monitor CPU, memory, I/O over long periods of time. - create/delete projects - create/delete apps - run builds - scale apps up/down - send traffic to apps - etc. From the log (linked in private followup comment), the panic occurred during a period of project deletion. NOTE: There are several issues in this log we are opening separate bz for. This particular issue can be seen by searching for this string: Observed a panic: &runtime.TypeAssertionError Version-Release number of selected component (if applicable): 3.7.0-0.178.0
https://github.com/openshift/origin/pull/17220
Verified on 3.7.5 - left a create projects -> create apps -> build apps -> delete projects hard loop running 24 hrs and no panics.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188