Bug 1995581
| Summary: | kube apiserver panic from admissions controller | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Stephen Benjamin <stbenjam> |
| Component: | kube-apiserver | Assignee: | Stephen Benjamin <stbenjam> |
| Status: | CLOSED DUPLICATE | QA Contact: | Ke Wang <kewang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.9 | CC: | aos-bugs, mfojtik, sippy, sttts, xxia |
| Target Milestone: | --- | ||
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-08-25 10:01:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Do you know since when this bug exists in kube? Do we have to backport? Just since 1.22 -- there's an upstream backport PR open: https://github.com/kubernetes/kubernetes/pull/104469 Will we update 1.22 again or should I open a PR to o/k? *** This bug has been marked as a duplicate of bug 1997465 *** |
Symptom Detection.Undiagnosed panic detected in pod pods/openshift-kube-apiserver_kube-apiserver-ci-op-62s95mxv-1368f-5s2lm-master-0_kube-apiserver_previous.log.gz:E0818 19:57:56.080795 17 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x4ad36a0), concrete:(*runtime._type)(nil), asserted:(*runtime._type)(0x4769e60), missingMethod:""} (interface conversion: interface {} is nil, not *golang_lru.entry) It is panicking here (see stack trace below) https://github.com/kubernetes/utils/blob/efc7438f0176a4a29c62bac8989d8dd183054fa5/internal/third_party/forked/golang/golang-lru/lru.go#L108 Following the stack trace , this is multi-threaded and I believe GetQuotas can be called multiple times by different go routines. https://github.com/kubernetes/apiserver/commit/fe1610f3fe33be47c3ce5f195abb30d3d5eb460b moved to using the LRU implementation from HashiCorp to https://github.com/kubernetes/utils/tree/master/internal/third_party/forked/golang The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See https://github.com/kubernetes/utils/blob/efc7438f0176a4a29c62bac8989d8dd183054fa5/internal/third_party/forked/golang/golang-lru/lru.go#L22 Stack trace: 2021-08-18T19:57:56.080933580Z E0818 19:57:56.080795 17 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x4ad36a0), concrete:(*runtime._type)(nil), asserted:(*runtime._type)(0x4769e60), missingMethod:""} (interface conversion: interface {} is nil, not *golang_lru.entry) 2021-08-18T19:57:56.080933580Z goroutine 5333 [running]: 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x4c78f00, 0xc0398de210) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86 2021-08-18T19:57:56.080933580Z panic(0x4c78f00, 0xc0398de210) 2021-08-18T19:57:56.080933580Z /usr/lib/golang/src/runtime/panic.go:965 +0x1b9 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru.(*Cache).removeElement(0xc00012b280, 0xc0013afef0) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru/lru.go:108 +0x169 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru.(*Cache).RemoveOldest(0xc00012b280) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru/lru.go:102 +0x53 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru.(*Cache).Add(0xc00012b280, 0x48992a0, 0xc02034c9f0, 0x4e9d700, 0xc0398de1b0) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/internal/third_party/forked/golang/golang-lru/lru.go:69 +0x330 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/utils/lru.(*Cache).Add(0xc00012b2a0, 0x48992a0, 0xc02034c9f0, 0x4e9d700, 0xc0398de1b0) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/lru/lru.go:43 +0x9b 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaAccessor).GetQuotas(0xc0010a9e00, 0xc0417ece83, 0xd, 0x1, 0x1, 0xc000c2e5d0, 0x1, 0x1) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/resource_access.go:130 +0x816 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).checkAttributes(0xc000b56c00, 0xc0417ece83, 0xd, 0xc05bdf8d98, 0x1, 0x1) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:179 +0xb4 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).doWork.func1(0xc009caf700) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:158 +0xf1 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).doWork(0xc000b56c00) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:162 +0x4b 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00ec818e0) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00ec818e0, 0x5e62440, 0xc00ecdc6f0, 0x1, 0xc00016a300) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00ec818e0, 0x3b9aca00, 0x0, 0x5810801, 0xc00016a300) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98 2021-08-18T19:57:56.080933580Z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00ec818e0, 0x3b9aca00, 0xc00016a300) 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d 2021-08-18T19:57:56.080933580Z created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota.(*quotaEvaluator).run 2021-08-18T19:57:56.080933580Z /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/resourcequota/controller.go:141 +0x96 2021-08-18T19:57:56.080933580Z E0818 19:57:56.080822 17 status.go:71] apiserver received an error that is not an metav1.Status: resourcequota.defaultDeny{}: DEFAULT DENY 2021-08-18T19:57:56.084173348Z panic: interface conversion: interface {} is nil, not *golang_lru.entry [recovered] 2021-08-18T19:57:56.084173348Z panic: interface conversion: interface {} is nil, not *golang_lru.entry