Symptom Detection.Undiagnosed panic detected in pod

pods/openshift-kube-apiserver_kube-apiserver-ci-op-62s95mxv-1368f-5s2lm-master-0_kube-apiserver_previous.log.gz:E0818 19:57:56.080795      17 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x4ad36a0), concrete:(*runtime._type)(nil), asserted:(*runtime._type)(0x4769e60), missingMethod:""} (interface conversion: interface {} is nil, not *golang_lru.entry)


It is panicking here (see stack trace below)

https://github.com/kubernetes/utils/blob/efc7438f0176a4a29c62bac8989d8dd183054fa5/internal/third_party/forked/golang/golang-lru/lru.go#L108

Following the stack trace , this is multi-threaded and I believe GetQuotas can be called multiple times by different go routines.

https://github.com/kubernetes/apiserver/commit/fe1610f3fe33be47c3ce5f195abb30d3d5eb460b moved to using the LRU implementation from HashiCorp to https://github.com/kubernetes/utils/tree/master/internal/third_party/forked/golang

The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See https://github.com/kubernetes/utils/blob/efc7438f0176a4a29c62bac8989d8dd183054fa5/internal/third_party/forked/golang/golang-lru/lru.go#L22



Stack trace:
2021-08-18T19:57:56.080933580Z E0818 19:57:56.080795      17 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x4ad36a0), concrete:(*runtime._type)(nil), asserted:(*runtime._type)(0x4769e60), missingMethod:""} (interface conversion: interface {} is nil, not *golang_lru.entry)
2021-08-18T19:57:56.080933580Z goroutine 5333 [running]:
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x4c78f00, 0xc0398de210)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
2021-08-18T19:57:56.080933580Z panic(0x4c78f00, 0xc0398de210)
2021-08-18T19:57:56.080933580Z  /usr/lib/golang/src/runtime/panic.go:965 +0x1b9
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru.(*Cache).removeElement(0xc00012b280, 0xc0013afef0)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru/lru.go:108 +0x169
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru.(*Cache).RemoveOldest(0xc00012b280)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru/lru.go:102 +0x53
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru.(*Cache).Add(0xc00012b280, 0x48992a0, 0xc02034c9f0, 0x4e9d700, 0xc0398de1b0)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru/lru.go:69 +0x330
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/lru.(*Cache).Add(0xc00012b2a0, 0x48992a0, 0xc02034c9f0, 0x4e9d700, 0xc0398de1b0)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/lru/lru.go:43 +0x9b
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaAccessor).GetQuotas(0xc0010a9e00, 0xc0417ece83, 0xd, 0x1, 0x1, 0xc000c2e5d0, 0x1, 0x1)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:130 +0x816
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).checkAttributes(0xc000b56c00, 0xc0417ece83, 0xd, 0xc05bdf8d98, 0x1, 0x1)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:179 +0xb4
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).doWork.func1(0xc009caf700)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:158 +0xf1
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).doWork(0xc000b56c00)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:162 +0x4b
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00ec818e0)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00ec818e0, 0x5e62440, 0xc00ecdc6f0, 0x1, 0xc00016a300)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00ec818e0, 0x3b9aca00, 0x0, 0x5810801, 0xc00016a300)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00ec818e0, 0x3b9aca00, 0xc00016a300)
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
2021-08-18T19:57:56.080933580Z created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).run
2021-08-18T19:57:56.080933580Z  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:141 +0x96
2021-08-18T19:57:56.080933580Z E0818 19:57:56.080822      17 status.go:71] apiserver received an error that is not an metav1.Status: resourcequota.defaultDeny{}: DEFAULT DENY
2021-08-18T19:57:56.084173348Z panic: interface conversion: interface {} is nil, not *golang_lru.entry [recovered]
2021-08-18T19:57:56.084173348Z  panic: interface conversion: interface {} is nil, not *golang_lru.entry