bump k8s.io/apiserver to 1.20.3: We need the following cherry pick PRs from upstream (but not limited to): https://github.com/kubernetes/kubernetes/pull/97862 https://github.com/kubernetes/kubernetes/pull/98183
This BZ will fix the following panic message that has already surfaced in CI (https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-openshift-tests-private-release-4.7-sanity/1351922587822723072) E0125 15:16:38.289815 1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference goroutine 224651 [running]: k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc007e935c0) k8s.io/apiserver.1/pkg/server/filters/timeout.go:106 +0x113 panic(0x2a0f120, 0x473beb0) runtime/panic.go:969 +0x1b9 k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc00052bb60, 0x338dda0, 0xc0079533e0, 0x33b4de0, 0xc0087bdd60, 0x2, 0x0, 0x0, 0x0, 0x0)
1.20.3 is going to be released on 2021-02-17, according to https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#cadence
Today hit same as comment 1's must-gather logs in UPI_vSphere_https_proxy_fips on_ovn env, one (and only one) of the oauth-apiserver pod instances hit: $ oc get clusterversion version 4.7.0-0.nightly-2021-02-02-164630 True False 30h Cluster version is 4.7.0-0.nightly-2021-02-02-164630 $ oc logs --timestamps apiserver-86bf876bc-b25qn -n openshift-oauth-apiserver | tee logs/oauth_apiserver-86bf876bc-b25qn.log 2021-02-04T02:45:37.360652417Z I0204 02:45:37.360617 1 trace.go:205] Trace[487475631]: "Get" url:/apis/oauth.openshift.io/v1/oauthaccesstokens/sha256~XcG9_XkhRlK5R6UgLJU5sVMVGEtntxZ4fa4GiPwm8A8,user-agent:kube-apiserver/v1.20.0+3d0efee (linux/amd64) kubernetes/3d0efee,client:::1 (04-Feb-2021 02:44:45.052) (total time: 52308ms): ... 2021-02-04T02:45:42.096669829Z E0204 02:45:42.096601 1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference 2021-02-04T02:45:42.096669829Z goroutine 778814 [running]: ... 2021-02-04T02:45:42.096669829Z k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc000a154a0, 0x2720b40, 0xc00083ea50, 0x273fbc0, 0xc002814c80, 0x2, 0xc001bf65f0, 0xeb51ef, 0x2721200, 0xc001aec7c0) ... 2021-02-04T02:45:42.096669829Z k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func1.4() 2021-02-04T02:45:42.096669829Z k8s.io/apiserver.1/pkg/server/filters/priority-and-fairness.go:127 +0x3c6 ... Extracted the rest of panic places: 2021-02-04T02:45:42.096743810Z E0204 02:45:42.096637 1 wrap.go:58] apiserver panic'd on GET /apis/user.openshift.io/v1/users/~ 2021-02-04T02:45:42.096760340Z E0204 02:45:42.096729 1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference 2021-02-04T02:45:42.096829089Z E0204 02:45:42.096762 1 wrap.go:58] apiserver panic'd on GET /apis/user.openshift.io/v1/users/~ 2021-02-04T02:45:42.096829089Z E0204 02:45:42.096762 1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference 2021-02-04T02:45:42.096870658Z E0204 02:45:42.096787 1 wrap.go:58] apiserver panic'd on GET /apis/user.openshift.io/v1/users/~ 2021-02-04T02:45:42.097690246Z E0204 02:45:42.097368 1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference Full log http://file.rdu.redhat.com/~xxia/bug/1917904/oauth_apiserver-86bf876bc-b25qn.log is uploaded.
lszaszki, > 2021-02-04T02:45:42.096669829Z k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc000a154a0, 0x2720b40, 0xc00083ea50, 0x273fbc0, 0xc002814c80, 0x2, 0xc001bf65f0, 0xeb51ef, 0x2721200, 0xc001aec7c0) ... 2021-02-04T02:45:42.096669829Z k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func1.4() 2021-02-04T02:45:42.096669829Z k8s.io/apiserver.1/pkg/server/filters/priority-and-fairness.go:127 +0x3c6 side note: should openshift-oauth-apiserver have p&f enabled?
The fix applies to both openshift-apiserver and oauth-apiserver
*** Bug 1917906 has been marked as a duplicate of this bug. ***
PRs are ready to be merged.
Using the verification approach in bug 1933599#c2 , check 4.7 jobs in last days since PR merged, did not find the previous encountered error in openshift-apiserver and oauth-apiserver now: https://search.ci.openshift.org/?search=apiserver.*Observed+a+panic%3A+runtime+error%3A+invalid+memory+address+or+nil+pointer+dereference&maxAge=168h&context=1&type=junit&name=4%5C.7&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job So moving to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.5 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1005