Bug 2047927 - 'oc get project' caused 'Observed a panic: cannot deep copy core.NamespacePhase' when AllRequestBodies is used
Summary: 'oc get project' caused 'Observed a panic: cannot deep copy core.NamespacePha...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-apiserver
Version: 4.10
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.10.0
Assignee: Abu Kashem
QA Contact: Rahul Gangwar
URL:
Whiteboard:
Depends On: 2047335
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-28 21:24 UTC by Abu Kashem
Modified: 2022-05-03 00:25 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2047335
Environment:
Last Closed: 2022-03-10 16:42:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-apiserver pull 273 0 None Merged Bug 2047927: [release-4.10] fix panic from printer 2022-02-02 14:56:04 UTC
Github openshift openshift-apiserver pull 278 0 None Merged [release-4.10] Bug 2047927: fix panic from printer 2022-02-03 18:12:05 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:43:04 UTC

Description Abu Kashem 2022-01-28 21:24:03 UTC
+++ This bug was initially created as a clone of Bug #2047335 +++

Description of problem:
'oc get project' caused 'Observed a panic: cannot deep copy core.NamespacePhase' when AllRequestBodies is used

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-26-234447

How reproducible:
Always

Steps to Reproduce:
1. oc edit apiserver cluster, change audit policy to AllRequestBodies.
2. Wait for pods to finish rotation
3. $ oc get project
Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get projects.project.openshift.io)

$ oc get project default
Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get projects.project.openshift.io default)

Actual results:
3. As above

Expected results:
3. Should succeed

Additional info:
OAS pod logs panic stack in next comment

--- Additional comment from Xingxing Xia on 2022-01-27 16:15:25 UTC ---

$ oc logs -n openshift-apiserver ONE_OAS_POD
...
2022-01-27T15:02:37.171735296Z E0127 15:02:37.171633       1 runtime.go:76] Observed a panic: cannot deep copy core.NamespacePhase
2022-01-27T15:02:37.171735296Z goroutine 3466 [running]:
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/timeout.go:105 +0xb0
2022-01-27T15:02:37.171735296Z panic({0x3c598a0, 0xc0020d9d10})
2022-01-27T15:02:37.171735296Z 	runtime/panic.go:1038 +0x215
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1.1·dwrap·1()
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/audit.go:86 +0x2a
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1.1()
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/audit.go:95 +0x250
2022-01-27T15:02:37.171735296Z panic({0x3c598a0, 0xc0020d9d10})
2022-01-27T15:02:37.171735296Z 	runtime/panic.go:1038 +0x215
2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue({0x3b8d100, 0xc0020d9c50})
2022-01-27T15:02:37.171735296Z 	k8s.io/apimachinery@v0.23.1/pkg/runtime/converter.go:639 +0x273
2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*TableRow).DeepCopy(0xc000592840)
2022-01-27T15:02:37.171735296Z 	k8s.io/apimachinery@v0.23.1/pkg/apis/meta/v1/deepcopy.go:33 +0xe8
2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*TableRow).DeepCopyInto(...)
2022-01-27T15:02:37.171735296Z 	k8s.io/apimachinery@v0.23.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1068
2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*Table).DeepCopyInto(0xc005435830, 0xc0054359e0)
2022-01-27T15:02:37.171735296Z 	k8s.io/apimachinery@v0.23.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1001 +0x27c
2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*Table).DeepCopy(...)
2022-01-27T15:02:37.171735296Z 	k8s.io/apimachinery@v0.23.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1013
2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*Table).DeepCopyObject(0xc005435830)
2022-01-27T15:02:37.171735296Z 	k8s.io/apimachinery@v0.23.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1019 +0x45
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/audit.copyWithoutManagedFields({0x46faf80, 0xc005435830})
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/audit/request.go:291 +0xb2
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/audit.LogResponseObject({0x4728f38, 0xc003981770}, {0x46faf80, 0xc005435830}, {{0xc005374d41, 0x0}, {0xc005374d3c, 0x0}}, {0x47164d8, 0xc0073c2980})
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/audit/request.go:210 +0x2da
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated({0x47164d8, 0xc0073c2980}, {0x4713c88, 0xc001efb520}, {{0xc005374d41, 0x40ff9db}, {0xc005374d3c, 0x40ff9db}}, {0x4716778, 0xc004d5da00}, ...)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/handlers/responsewriters/writers.go:271 +0x3ff
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject({0x4728f38, 0xc003981980}, 0xc001efb520, 0x4, _, {_, _}, _, {0x0, {0x0, ...}, ...}, ...)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/handlers/response.go:137 +0x565
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/handlers.getResourceHandler.func1({0x4716778, 0xc004d5da00}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/handlers/get.go:79 +0x7fc
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints.restfulGetResource.func1(0xc0039818c0, 0xc000c1bc00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/installer.go:1221 +0x5b
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc0039818c0, 0xc000c1bc00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/metrics/metrics.go:509 +0x223
2022-01-27T15:02:37.171735296Z github.com/emicklei/go-restful.(*Container).dispatch(0xc0005b5560, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:288 +0x9d1
2022-01-27T15:02:37.171735296Z github.com/emicklei/go-restful.(*Container).Dispatch(...)
2022-01-27T15:02:37.171735296Z 	github.com/emicklei/go-restful@v2.9.5+incompatible/container.go:199
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4124941, 0x203001}, 0xc0005b5560, 0xc000755f80}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:146 +0x5bb
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc003423040, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:254 +0x385
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:234 +0x70
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x411eed5, 0x203001}, 0xc0036577a0, 0xc000768380}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:154 +0x75c
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc002172b00, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:254 +0x385
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:234 +0x70
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x411efed, 0x203001}, 0xc00200eb40, 0xc00036d0a0}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:154 +0x75c
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc004724980, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:254 +0x385
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:234 +0x70
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4127dd7, 0x203001}, 0xc0018830e0, 0xc000670c40}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:154 +0x75c
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc000890000, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:254 +0x385
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:234 +0x70
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4127f2c, 0xc00312a4f0}, 0xc0005b4d80, 0xc000349960}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:154 +0x75c
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc005d5ed40, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:254 +0x385
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/mux/pathrecorder.go:234 +0x70
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4107c93, 0x469a9b8}, 0xc0005b53b0, 0xc000869f80}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:154 +0x75c
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithWebhookDuration.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/webhook_duration.go:31 +0x24e
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc007356ad0, {0x4716778, 0xc004d5d880}, 0x1c903ea)
2022-01-27T15:02:37.171735296Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc000597b80, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/authorization.go:64 +0x425
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0x64d6108, {0x4716778, 0xc004d5d880}, 0x0)
2022-01-27T15:02:37.171735296Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0x6372760, {0x4716778, 0xc004d5d880}, 0x3)
2022-01-27T15:02:37.171735296Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/maxinflight.go:187 +0x29d
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc007356ef8, {0x4716778, 0xc004d5d880}, 0x1c903ea)
2022-01-27T15:02:37.171735296Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc0073c1200, {0x4716778, 0xc004d5d880}, 0xc0073c0fc0)
2022-01-27T15:02:37.171735296Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171735296Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/impersonation.go:50 +0x21c
2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0x64d6108, {2022-01-27T15:02:37.171824805Z 0x4716778, 0xc004d5d880}, 0xa)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc0073577b8, {0x4716778, 0xc004d5d880}, 0x1c903ea)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4729de0, {0x4716778, 0xc004d5d880}, 0xc000881b80)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/audit.go:117 +0x713
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x64d6108, {0x4716778, 0xc004d5d820}, 0xa)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc003981560, {0x4716778, 0xc004d5d820}, 0x1c903ea)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4716778, 0xc004d5d820}, 0x42b2e88)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/authentication.go:80 +0x839
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f00, {0x4716778, 0xc004d5d820}, 0x469a9b8)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d820}, 0xc0073b5800)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filterlatency/filterlatency.go:89 +0x46b
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc0045b7e00, {0x4716778, 0xc004d5d820}, 0x19f0912)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/timeout.go:110 +0x70
2022-01-27T15:02:37.171824805Z created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/timeout.go:96 +0x1bd
2022-01-27T15:02:37.171824805Z 
2022-01-27T15:02:37.171824805Z goroutine 3447 [running]:
2022-01-27T15:02:37.171824805Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3b8f000, 0xc0020d9e00})
2022-01-27T15:02:37.171824805Z 	k8s.io/apimachinery@v0.23.1/pkg/util/runtime/runtime.go:74 +0x7d
2022-01-27T15:02:37.171824805Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc00068c3e8, 0x1, 0x29396e6})
2022-01-27T15:02:37.171824805Z 	k8s.io/apimachinery@v0.23.1/pkg/util/runtime/runtime.go:48 +0x75
2022-01-27T15:02:37.171824805Z panic({0x3b8f000, 0xc0020d9e00})
2022-01-27T15:02:37.171824805Z 	runtime/panic.go:1038 +0x215
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0016c75d8, {0x4708a98, 0xc000012e88}, 0xdf8475800)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/timeout.go:116 +0x337
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x4708a98, 0xc000012e88}, 0xc0073b5700)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/request_deadline.go:100 +0x494
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc0073b5700, {0x4708a98, 0xc000012e88}, 0x100000000000000)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x4708a98, 0xc000012e88}, 0xc0073b5700)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/waitgroup.go:77 +0x766
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0x88)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithAuditAnnotations.func1({0x4708a98, 0xc000012e88}, 0xc0073b5600)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/audit_annotations.go:36 +0x230
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0x469a9b8)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x4708a98, 0xc000012e88}, 0xc0073b5500)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/warning.go:35 +0x2bb
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3fa4aa0, {0x4708a98, 0xc000012e88}, 0xd)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x4708a98, 0xc000012e88}, 0x3)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/cachecontrol.go:31 +0x126
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x203001, {0x4708a98, 0xc000012e88}, 0x30)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x4708a98, 0xc000012e88}, 0xc0073b5500)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/httplog/httplog.go:108 +0xaf
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0x469a9b8)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x4708a98, 0xc000012e88}, 0xc0073b5400)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/requestinfo.go:39 +0x316
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0xc0050e78f0)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x4708a98, 0xc000012e88}, 0xc0073b5300)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/request_received_time.go:38 +0x27e
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x1476d22, {0x4708a98, 0xc000012e88}, 0x0)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x4708a98, 0xc000012e88}, 0xc0073b5300)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x2c2
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3c91dc0, {0x4708a98, 0xc000012e88}, 0x100000000000008)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x4708a98, 0xc000012e88}, 0xc005359e40)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/filters/wrap.go:74 +0xba
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3eb80c0, {0x4708a98, 0xc000012e88}, 0x8)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x4708a98, 0xc000012e88}, 0xc00893e100)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/endpoints/filters/with_auditid.go:66 +0x40d
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3eb80c0, {0x4708a98, 0xc000012e88}, 0xd)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z github.com/openshift/library-go/pkg/apiserver/apiserverconfig.WithCacheControl.func1({0x4708a98, 0xc000012e88}, 0xc00893e100)
2022-01-27T15:02:37.171824805Z 	github.com/openshift/library-go@v0.0.0-20211216141749-1816407208cf/pkg/apiserver/apiserverconfig/cachecontrol.go:33 +0x1bd
2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc001adbca0, {0x4708a98, 0xc000012e88}, 0x24)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2046 +0x2f
2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0, {0x4708a98, 0xc000012e88}, 0x4728ec8)
2022-01-27T15:02:37.171824805Z 	k8s.io/apiserver@v0.23.1/pkg/server/handler.go:189 +0x2b
2022-01-27T15:02:37.171824805Z net/http.serverHandler.ServeHTTP({0x0}, {0x4708a98, 0xc000012e88}, 0xc00893e100)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:2878 +0x43b
2022-01-27T15:02:37.171824805Z net/http.initALPNRequest.ServeHTTP({{0x4728f38, 0xc00339bcb0}, 0xc003dc9880, {0xc004a321c0}}, {0x4708a98, 0xc000012e88}, 0xc00893e100)
2022-01-27T15:02:37.171824805Z 	net/http/server.go:3479 +0x245
2022-01-27T15:02:37.171824805Z golang.org/x/net/http2.(*serverConn).runHandler(0x4700ef8, 0x64d6108, 0x0, 0x0)
2022-01-27T15:02:37.171824805Z 	golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/server.go:2184 +0x78
2022-01-27T15:02:37.171824805Z created by golang.org/x/net/http2.(*serverConn).processHeaders
2022-01-27T15:02:37.171824805Z 	golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/server.go:1914 +0x590
2022-01-27T15:02:37.171824805Z E0127 15:02:37.171716       1 wrap.go:58] "apiserver panic'd" method="GET" URI="/apis/project.openshift.io/v1/projects/default" audit-ID="fbb1a4ce-23f9-4a5e-89ba-5e7cc1420a48"
...

--- Additional comment from Abu Kashem on 2022-01-27 21:56:02 UTC ---

xxia@redhat.com,
can you attach the log file with this error?

--- Additional comment from Abu Kashem on 2022-01-27 21:56:56 UTC ---

also, do you see similar error in kube-apiserver?

--- Additional comment from Xingxing Xia on 2022-01-28 02:06:47 UTC ---

Attaching the log file.
I ran below OAS resources: oc get project, oc new-project test-proj, oc delete project test-proj, oc get route, oc get build, only 'oc get project' triggered the panic.
I ran KAS resources like oc get pod, oc get ns, no such issue.
I ran OauthAS resources like oc get user, oc get identity, no such issue.

Then I tested WriteRequestBodies, 'oc get project' does not have such issue.

Then I tested latest 4.9 AllRequestBodies, no such issue, so it is a 4.10 regression, adding the Keywords.

--- Additional comment from Abu Kashem on 2022-01-28 19:36:59 UTC ---

xxia@redhat.com

Thanks for the information, I tracked it down, I think openshift/openshift-apiserver/pull/272 should fix the issue. Please verify when you have the next build

Comment 1 Abu Kashem 2022-01-28 21:25:31 UTC
> oc get projects will panic if audit policy is set to AllRequestBodies

It should be a blocker+ for 4.10

Comment 2 Abu Kashem 2022-01-31 13:23:38 UTC
xxia@redhat.com,

can you verify that every openshift API works as expected (no panic)

Comment 3 Abu Kashem 2022-01-31 13:55:16 UTC
xxia@redhat.com,
can you confirm that `oc get scc` also causes a panic with the current build?

Comment 5 Rahul Gangwar 2022-02-01 12:53:30 UTC
 oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.ci-2022-02-01-092557   True        False         16m     Cluster version is 4.10.0-0.ci-2022-02-01-092557

[rgangwar@preserver-workloadrhel-1 ~]$ oc get pod -n openshift-kube-apiserver -l apiserver --show-labels
NAME                                                                   READY   STATUS    RESTARTS   AGE   LABELS
kube-apiserver-rgangwar-01de2-dt8bf-master-0.c.openshift-qe.internal   5/5     Running   0          21m   apiserver=true,app=openshift-kube-apiserver,revision=5
kube-apiserver-rgangwar-01de2-dt8bf-master-1.c.openshift-qe.internal   5/5     Running   0          18m   apiserver=true,app=openshift-kube-apiserver,revision=5
kube-apiserver-rgangwar-01de2-dt8bf-master-2.c.openshift-qe.internal   5/5     Running   0          20m   apiserver=true,app=openshift-kube-apiserver,revision=5

[rgangwar@preserver-workloadrhel-1 ~]$ oc edit apiserver/cluster
apiserver.config.openshift.io/cluster edited

oc get pod -n openshift-kube-apiserver -l apiserver --show-labels 
NAME                                                                   READY   STATUS    RESTARTS   AGE     LABELS
kube-apiserver-rgangwar-01de2-dt8bf-master-0.c.openshift-qe.internal   5/5     Running   0          3m42s   apiserver=true,app=openshift-kube-apiserver,revision=6
kube-apiserver-rgangwar-01de2-dt8bf-master-1.c.openshift-qe.internal   5/5     Running   0          5m56s   apiserver=true,app=openshift-kube-apiserver,revision=6
kube-apiserver-rgangwar-01de2-dt8bf-master-2.c.openshift-qe.internal   5/5     Running   0          86s     apiserver=true,app=openshift-kube-apiserver,revision=6

oc get apiserver/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    oauth-apiserver.openshift.io/secure-token-storage: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-02-01T11:52:32Z"
  generation: 2
  name: cluster
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    name: version
    uid: 0dde0018-5415-4e3b-8c61-5a3371429bff
  resourceVersion: "33779"
  uid: 14d90180-42fd-420d-bcf4-b3a22eb74ac1
spec:
  audit:
    profile: AllRequestBodies

oc get project default
NAME      DISPLAY NAME   STATUS
default                  Active
[rgangwar@preserver-workloadrhel-1 ~]$ oc get projects
NAME                                               DISPLAY NAME   STATUS
default                                                           Active
kube-node-lease                                                   Active

Comment 6 Abu Kashem 2022-02-01 14:04:43 UTC
moving it back to assigned since we found two more cases of panic with SCC and Templates - once the master PR https://github.com/openshift/openshift-apiserver/pull/275 merges we will need to back port to 4.10

Comment 10 Rahul Gangwar 2022-02-04 08:04:07 UTC
 oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-02-04-015640   True        False         34m     Cluster version is 4.10.0-0.nightly-2022-02-04-015640

rahulgangwar@rgangwar-mac ~ % oc get pod -n openshift-kube-apiserver -l apiserver --show-labels
NAME                                                                   READY   STATUS    RESTARTS   AGE   LABELS
kube-apiserver-rgangwar-04de0-t8zdz-master-0.c.openshift-qe.internal   5/5     Running   0          38m   apiserver=true,app=openshift-kube-apiserver,revision=8
kube-apiserver-rgangwar-04de0-t8zdz-master-1.c.openshift-qe.internal   5/5     Running   0          39m   apiserver=true,app=openshift-kube-apiserver,revision=8
kube-apiserver-rgangwar-04de0-t8zdz-master-2.c.openshift-qe.internal   5/5     Running   0          36m   apiserver=true,app=openshift-kube-apiserver,revision=8

rahulgangwar@rgangwar-mac ~ % oc edit apiserver/cluster
apiserver.config.openshift.io/cluster edited

 oc get pod -n openshift-kube-apiserver -l apiserver --show-labels   
NAME                                                                   READY   STATUS    RESTARTS   AGE   LABELS
kube-apiserver-rgangwar-04de0-t8zdz-master-0.c.openshift-qe.internal   5/5     Running   0          10m   apiserver=true,app=openshift-kube-apiserver,revision=9
kube-apiserver-rgangwar-04de0-t8zdz-master-1.c.openshift-qe.internal   5/5     Running   0          12m   apiserver=true,app=openshift-kube-apiserver,revision=9
kube-apiserver-rgangwar-04de0-t8zdz-master-2.c.openshift-qe.internal   5/5     Running   0          15m   apiserver=true,app=openshift-kube-apiserver,revision=9

 oc get project default
NAME      DISPLAY NAME   STATUS
default                  Active

 oc get templates -n openshift|head
NAME                                            DESCRIPTION                                                                        PARAMETERS        OBJECTS
3scale-gateway                                  3scale's APIcast is an NGINX based API gateway used to integrate your interna...   17 (8 blank)      3
amq63-basic                                     Application template for JBoss A-MQ brokers. These can be deployed as standal...   11 (4 blank)      6
amq63-persistent                                An example JBoss A-MQ application. For more information about using this temp...   13 (4 blank)      8
amq63-persistent-ssl                            An example JBoss A-MQ application. For more information about using this temp...   18 (6 blank)      12
amq63-ssl                                       An example JBoss A-MQ application. For more information about using this temp...   16 (6 blank)      10
apicurito                                       Design beautiful, functional APIs with zero coding, using a visual designer f...   7 (1 blank)       7
cache-service                                   Red Hat Data Grid is an in-memory, distributed key/value store.                    8 (1 blank)       4
cakephp-mysql-example                           An example CakePHP application with a MySQL database. For more information ab...   21 (4 blank)      8
cakephp-mysql-persistent                        An example CakePHP application with a MySQL database. For more information ab...   22 (4 blank)  
    9

cat scc.yaml 
kind: SecurityContextConstraints
apiVersion: v1
metadata:
  name: scc-admin
allowPrivilegedContainer: true
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
fsGroup:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
users:
- my-admin-user
groups:
- my-admin-group


oc create -f scc.yaml 
W0204 13:31:44.213822   72388 shim_kubectl.go:55] Using non-groupfied API resources is deprecated and will be removed in a future release, update apiVersion to "security.openshift.io/v1" for your resource
securitycontextconstraints.security.openshift.io/scc-admin created

oc get scc scc-admin 
NAME        PRIV   CAPS         SELINUX    RUNASUSER   FSGROUP    SUPGROUP   PRIORITY     READONLYROOTFS   VOLUMES
scc-admin   true   <no value>   RunAsAny   RunAsAny    RunAsAny   RunAsAny   <no value>   false            ["awsElasticBlockStore","azureDisk","azureFile","cephFS","cinder","configMap","csi","downwardAPI","emptyDir","ephemeral","fc","flexVolume","flocker","gcePersistentDisk","gitRepo","glusterfs","iscsi","nfs","persistentVolumeClaim","photonPersistentDisk","portworxVolume","projected","quobyte","rbd","scaleIO","secret","storageOS","vsphere"]

Comment 13 errata-xmlrpc 2022-03-10 16:42:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.