Bug 1907390 - prometheus-adapter: panic after k8s 1.20 bump
Summary: prometheus-adapter: panic after k8s 1.20 bump
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.7
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.7.0
Assignee: Sergiusz Urbaniak
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-14 12:26 UTC by Sergiusz Urbaniak
Modified: 2021-02-24 15:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:43:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift k8s-prometheus-adapter pull 41 0 None closed Bug 1907390: bump to k8s 1.20 2021-01-25 09:01:23 UTC
Red Hat Bugzilla 1907373 0 urgent CLOSED Rebase to kube 1.20.0 2021-02-24 15:43:41 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:43:59 UTC

Description Sergiusz Urbaniak 2020-12-14 12:26:43 UTC
As discussed in https://coreos.slack.com/archives/C0VMT03S5/p1607944537414500, and observed in https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_kubernetes/471/pull-ci-openshift-kubernetes-master-e2e-aws-selfupgrade/1338383564957290496 we see a panic regular in prometheus-adapter:

2020-12-14T09:33:36.201098552Z E1214 09:33:32.333299       1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing connection/stream because serving request timed out and response had been started)
2020-12-14T09:33:36.201098552Z goroutine 2209 [running]:
2020-12-14T09:33:36.201098552Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1a79a80, 0xc000377320)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
2020-12-14T09:33:36.201098552Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc001337c98, 0x1, 0x1)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
2020-12-14T09:33:36.201098552Z panic(0x1a79a80, 0xc000377320)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/runtime/panic.go:969 +0x175
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc000836e20, 0xc000b168c0)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1bc
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc00026a560, 0x1fb1340, 0xc0004e12d0, 0xc001566a00)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x2f3
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x1fb1340, 0xc0004e12d0, 0xc001303400)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:59 +0x137
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc0006b6090, 0x1fb1340, 0xc0004e12d0, 0xc001303400)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x1fb1340, 0xc0004e12d0, 0xc001303300)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x269
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc0006b60c0, 0x1fb1340, 0xc0004e12d0, 0xc001303300)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1(0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x1a7
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a620, 0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a680, 0x1fb1340, 0xc0004e12d0, 0xc001303200)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:91 +0x2f2
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a6c0, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0xe6
2020-12-14T09:33:36.201098552Z net/http.HandlerFunc.ServeHTTP(0xc00026a6e0, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2054 +0x44
2020-12-14T09:33:36.201098552Z k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc0006b60f0, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
2020-12-14T09:33:36.201098552Z net/http.serverHandler.ServeHTTP(0xc000550460, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:2855 +0xa3
2020-12-14T09:33:36.201098552Z net/http.initALPNRequest.ServeHTTP(0x1fb65c0, 0xc0019e2270, 0xc000865180, 0xc000550460, 0x1fa4a00, 0xc000010240, 0xc001302700)
2020-12-14T09:33:36.201098552Z 	/usr/lib/golang/src/net/http/server.go:3427 +0x8d
2020-12-14T09:33:36.201098552Z golang.org/x/net/http2.(*serverConn).runHandler(0xc0000b7080, 0xc000010240, 0xc001302700, 0xc0017fa3a0)
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:2147 +0x8b
2020-12-14T09:33:36.201098552Z created by golang.org/x/net/http2.(*serverConn).processHeaders
2020-12-14T09:33:36.201098552Z 	/go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/golang.org/x/net/http2/server.go:1881 +0x505
2020-12-14T09:33:36.588214030Z E1214 09:33:36.556420       1 webhook.go:199] Failed to make webhook authorizer request: context canceled

Comment 2 Junqi Zhao 2020-12-18 03:00:21 UTC
checked with 4.7.0-0.nightly-2020-12-17-201522, fix is in payload and did not see the panic error for prometheus-adapter, k8s version is v1.20
# oc get node
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-132-177.us-east-2.compute.internal   Ready    master   62m   v1.20.0+87544c5
..

Comment 6 errata-xmlrpc 2021-02-24 15:43:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.