As discussed in https://coreos.slack.com/archives/C0VMT03S5/p1607944537414500, and observed in https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_kubernetes/471/pull-ci-openshift-kubernetes-master-e2e-aws-selfupgrade/1338383564957290496 we see a panic regular in prometheus-adapter: 2020-12-14T09:33:36.201098552Z E1214 09:33:32.333299 1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing connection/stream because serving request timed out and response had been started) 2020-12-14T09:33:36.201098552Z goroutine 2209 [running]: 2020-12-14T09:33:36.201098552Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1a79a80, 0xc000377320) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6 2020-12-14T09:33:36.201098552Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc001337c98, 0x1, 0x1) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89 2020-12-14T09:33:36.201098552Z panic(0x1a79a80, 0xc000377320) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/runtime/panic.go:969 +0x175 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc000836e20, 0xc000b168c0) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1bc 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00026a560, 0x1fb1340, 0xc0004e12d0, 0xc001566a00) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x2f3 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x1fb1340, 0xc0004e12d0, 0xc001303400) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137 2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc0006b6090, 0x1fb1340, 0xc0004e12d0, 0xc001303400) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2054 +0x44 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x1fb1340, 0xc0004e12d0, 0xc001303300) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269 2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc0006b60c0, 0x1fb1340, 0xc0004e12d0, 0xc001303300) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2054 +0x44 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x1fb1340, 0xc0004e12d0, 0xc001303200) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a7 2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a620, 0x1fb1340, 0xc0004e12d0, 0xc001303200) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2054 +0x44 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x1fb1340, 0xc0004e12d0, 0xc001303200) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8 2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a680, 0x1fb1340, 0xc0004e12d0, 0xc001303200) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2054 +0x44 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2f2 2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a6c0, 0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2054 +0x44 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6 2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a6e0, 0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2054 +0x44 2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc0006b60f0, 0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51 2020-12-14T09:33:36.201098552Z net/http.serverHandler.ServeHTTP(0xc000550460, 0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:2855 +0xa3 2020-12-14T09:33:36.201098552Z net/http.initALPNRequest.ServeHTTP(0x1fb65c0, 0xc0019e2270, 0xc000865180, 0xc000550460, 0x1fa4a00, 0xc000010240, 0xc001302700) 2020-12-14T09:33:36.201098552Z /usr/lib/golang/src/net/http/server.go:3427 +0x8d 2020-12-14T09:33:36.201098552Z golang.org/x/net/http2.(*serverConn).runHandler(0xc0000b7080, 0xc000010240, 0xc001302700, 0xc0017fa3a0) 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:2147 +0x8b 2020-12-14T09:33:36.201098552Z created by golang.org/x/net/http2.(*serverConn).processHeaders 2020-12-14T09:33:36.201098552Z /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:1881 +0x505 2020-12-14T09:33:36.588214030Z E1214 09:33:36.556420 1 webhook.go:199] Failed to make webhook authorizer request: context canceled
checked with 4.7.0-0.nightly-2020-12-17-201522, fix is in payload and did not see the panic error for prometheus-adapter, k8s version is v1.20 # oc get node NAME STATUS ROLES AGE VERSION ip-10-0-132-177.us-east-2.compute.internal Ready master 62m v1.20.0+87544c5 ..
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633