Bug 1917904 - [release-4.7] bump k8s.io/apiserver to 1.20.3
Summary: [release-4.7] bump k8s.io/apiserver to 1.20.3
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-apiserver
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.z
Assignee: Lukasz Szaszkiewicz
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On: 1933599
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-19 16:26 UTC by Abu Kashem
Modified: 2021-04-05 13:56 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1933599 (view as bug list)
Environment:
Last Closed: 2021-04-05 13:55:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oauth-apiserver pull 43 0 None open Bug 1917904: bump k8s.io/apiserver to 1.20.3 2021-02-22 10:47:47 UTC
Github openshift openshift-apiserver pull 187 0 None open Bug 1917904: bump k8s.io/apiserver to 1.20.3 2021-02-22 10:46:08 UTC
Red Hat Product Errata RHSA-2021:1005 0 None None None 2021-04-05 13:56:09 UTC

Description Abu Kashem 2021-01-19 16:26:51 UTC
bump k8s.io/apiserver to 1.20.3:

We need the following cherry pick PRs from upstream (but not limited to):
https://github.com/kubernetes/kubernetes/pull/97862
https://github.com/kubernetes/kubernetes/pull/98183

Comment 1 Lukasz Szaszkiewicz 2021-01-26 12:53:08 UTC
This BZ will fix the following panic message that has already surfaced in CI (https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-openshift-tests-private-release-4.7-sanity/1351922587822723072)


E0125 15:16:38.289815       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
goroutine 224651 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc007e935c0)
	k8s.io/apiserver.1/pkg/server/filters/timeout.go:106 +0x113
panic(0x2a0f120, 0x473beb0)
	runtime/panic.go:969 +0x1b9
k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc00052bb60, 0x338dda0, 0xc0079533e0, 0x33b4de0, 0xc0087bdd60, 0x2, 0x0, 0x0, 0x0, 0x0)

Comment 2 Lukasz Szaszkiewicz 2021-01-26 13:02:55 UTC
1.20.3 is going to be released on 2021-02-17, according to https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#cadence

Comment 4 Xingxing Xia 2021-02-04 09:32:56 UTC
Today hit same as comment 1's must-gather logs in UPI_vSphere_https_proxy_fips on_ovn env, one (and only one) of the oauth-apiserver pod instances hit:
$ oc get clusterversion
version   4.7.0-0.nightly-2021-02-02-164630   True        False         30h     Cluster version is 4.7.0-0.nightly-2021-02-02-164630

$ oc logs --timestamps apiserver-86bf876bc-b25qn -n openshift-oauth-apiserver | tee logs/oauth_apiserver-86bf876bc-b25qn.log
2021-02-04T02:45:37.360652417Z I0204 02:45:37.360617       1 trace.go:205] Trace[487475631]: "Get" url:/apis/oauth.openshift.io/v1/oauthaccesstokens/sha256~XcG9_XkhRlK5R6UgLJU5sVMVGEtntxZ4fa4GiPwm8A8,user-agent:kube-apiserver/v1.20.0+3d0efee (linux/amd64) kubernetes/3d0efee,client:::1 (04-Feb-2021 02:44:45.052) (total time: 52308ms):
...
2021-02-04T02:45:42.096669829Z E0204 02:45:42.096601       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
2021-02-04T02:45:42.096669829Z goroutine 778814 [running]:
...
2021-02-04T02:45:42.096669829Z k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc000a154a0, 0x2720b40, 0xc00083ea50, 0x273fbc0, 0xc002814c80, 0x2, 0xc001bf65f0, 0xeb51ef, 0x2721200, 0xc001aec7c0)
...
2021-02-04T02:45:42.096669829Z k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func1.4()
2021-02-04T02:45:42.096669829Z  k8s.io/apiserver.1/pkg/server/filters/priority-and-fairness.go:127 +0x3c6
...
Extracted the rest of panic places:
2021-02-04T02:45:42.096743810Z E0204 02:45:42.096637       1 wrap.go:58] apiserver panic'd on GET /apis/user.openshift.io/v1/users/~
2021-02-04T02:45:42.096760340Z E0204 02:45:42.096729       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference

2021-02-04T02:45:42.096829089Z E0204 02:45:42.096762       1 wrap.go:58] apiserver panic'd on GET /apis/user.openshift.io/v1/users/~
2021-02-04T02:45:42.096829089Z E0204 02:45:42.096762       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference

2021-02-04T02:45:42.096870658Z E0204 02:45:42.096787       1 wrap.go:58] apiserver panic'd on GET /apis/user.openshift.io/v1/users/~
2021-02-04T02:45:42.097690246Z E0204 02:45:42.097368       1 runtime.go:76] Observed a panic: runtime error: invalid memory address or nil pointer dereference
Full log http://file.rdu.redhat.com/~xxia/bug/1917904/oauth_apiserver-86bf876bc-b25qn.log is uploaded.

Comment 5 Abu Kashem 2021-02-04 15:28:50 UTC
lszaszki,

> 2021-02-04T02:45:42.096669829Z k8s.io/apiserver/plugin/pkg/authorizer/webhook.(*WebhookAuthorizer).Authorize(0xc000a154a0, 0x2720b40, 0xc00083ea50, 0x273fbc0, 0xc002814c80, 0x2, 0xc001bf65f0, 0xeb51ef, 0x2721200, 0xc001aec7c0)
...
2021-02-04T02:45:42.096669829Z k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func1.4()
2021-02-04T02:45:42.096669829Z  k8s.io/apiserver.1/pkg/server/filters/priority-and-fairness.go:127 +0x3c6

side note: should openshift-oauth-apiserver have p&f enabled?

Comment 8 Lukasz Szaszkiewicz 2021-02-22 10:48:14 UTC
The fix applies to both openshift-apiserver and oauth-apiserver

Comment 10 Standa Laznicka 2021-03-08 09:57:27 UTC
*** Bug 1917906 has been marked as a duplicate of this bug. ***

Comment 11 Lukasz Szaszkiewicz 2021-03-19 16:21:01 UTC
PRs are ready to be merged.

Comment 14 Xingxing Xia 2021-03-29 10:38:54 UTC
Using the verification approach in bug 1933599#c2 , check 4.7 jobs in last days since PR merged, did not find the previous encountered error in openshift-apiserver and oauth-apiserver now:
https://search.ci.openshift.org/?search=apiserver.*Observed+a+panic%3A+runtime+error%3A+invalid+memory+address+or+nil+pointer+dereference&maxAge=168h&context=1&type=junit&name=4%5C.7&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
So moving to VERIFIED.

Comment 16 errata-xmlrpc 2021-04-05 13:55:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.5 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1005


Note You need to log in before you can comment on or make changes to this bug.