Bug 1907390

Summary:	prometheus-adapter: panic after k8s 1.20 bump
Product:	OpenShift Container Platform	Reporter:	Sergiusz Urbaniak <surbania>
Component:	Monitoring	Assignee:	Sergiusz Urbaniak <surbania>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.7	CC:	alegrand, anpicker, erooth, kakkoyun, lcosic, pkrupa, surbania
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-24 15:43:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sergiusz Urbaniak 2020-12-14 12:26:43 UTC

As discussed in https://coreos.slack.com/archives/C0VMT03S5/p1607944537414500, and observed in https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_kubernetes/471/pull-ci-openshift-kubernetes-master-e2e-aws-selfupgrade/1338383564957290496 we see a panic regular in prometheus-adapter:

2020-12-14T09:33:36.201098552Z E1214 09:33:32.333299       1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing connection/stream because serving request timed out and response had been started)
2020-12-14T09:33:36.201098552Z goroutine 2209 [running]:
2020-12-14T09:33:36.201098552Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1a79a80, 0xc000377320)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
2020-12-14T09:33:36.201098552Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc001337c98, 0x1, 0x1)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
2020-12-14T09:33:36.201098552Z panic(0x1a79a80, 0xc000377320)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/runtime/panic.go:969 +0x175
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc000836e20, 0xc000b168c0)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1bc
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00026a560, 0x1fb1340, 0xc0004e12d0, 0xc001566a00)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x2f3
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x1fb1340, 0xc0004e12d0, 0xc001303400)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc0006b6090, 0x1fb1340, 0xc0004e12d0, 0xc001303400)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x1fb1340, 0xc0004e12d0, 0xc001303300)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc0006b60c0, 0x1fb1340, 0xc0004e12d0, 0xc001303300)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a7
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a620, 0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a680, 0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2f2
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a6c0, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a6e0, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc0006b60f0, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
2020-12-14T09:33:36.201098552Z net/http.serverHandler.ServeHTTP(0xc000550460, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2855 +0xa3
2020-12-14T09:33:36.201098552Z net/http.initALPNRequest.ServeHTTP(0x1fb65c0, 0xc0019e2270, 0xc000865180, 0xc000550460, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:3427 +0x8d
2020-12-14T09:33:36.201098552Z golang.org/x/net/http2.(*serverConn).runHandler(0xc0000b7080, 0xc000010240, 0xc001302700, 0xc0017fa3a0)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:2147 +0x8b
2020-12-14T09:33:36.201098552Z created by golang.org/x/net/http2.(*serverConn).processHeaders
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:1881 +0x505
2020-12-14T09:33:36.588214030Z E1214 09:33:36.556420       1 webhook.go:199] Failed to make webhook authorizer request: context canceled

Comment 2 Junqi Zhao 2020-12-18 03:00:21 UTC

checked with 4.7.0-0.nightly-2020-12-17-201522, fix is in payload and did not see the panic error for prometheus-adapter, k8s version is v1.20
# oc get node
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-132-177.us-east-2.compute.internal   Ready    master   62m   v1.20.0+87544c5
..

Comment 6 errata-xmlrpc 2021-02-24 15:43:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633