+++ This bug was initially created as a clone of Bug #2047335 +++ Description of problem: 'oc get project' caused 'Observed a panic: cannot deep copy core.NamespacePhase' when AllRequestBodies is used Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-26-234447 How reproducible: Always Steps to Reproduce: 1. oc edit apiserver cluster, change audit policy to AllRequestBodies. 2. Wait for pods to finish rotation 3. $ oc get project Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get projects.project.openshift.io) $ oc get project default Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get projects.project.openshift.io default) Actual results: 3. As above Expected results: 3. Should succeed Additional info: OAS pod logs panic stack in next comment --- Additional comment from Xingxing Xia on 2022-01-27 16:15:25 UTC --- $ oc logs -n openshift-apiserver ONE_OAS_POD ... 2022-01-27T15:02:37.171735296Z E0127 15:02:37.171633 1 runtime.go:76] Observed a panic: cannot deep copy core.NamespacePhase 2022-01-27T15:02:37.171735296Z goroutine 3466 [running]: 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1() 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/filters/timeout.go:105 +0xb0 2022-01-27T15:02:37.171735296Z panic({0x3c598a0, 0xc0020d9d10}) 2022-01-27T15:02:37.171735296Z runtime/panic.go:1038 +0x215 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1.1·dwrap·1() 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filters/audit.go:86 +0x2a 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1.1() 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filters/audit.go:95 +0x250 2022-01-27T15:02:37.171735296Z panic({0x3c598a0, 0xc0020d9d10}) 2022-01-27T15:02:37.171735296Z runtime/panic.go:1038 +0x215 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue({0x3b8d100, 0xc0020d9c50}) 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery.1/pkg/runtime/converter.go:639 +0x273 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*TableRow).DeepCopy(0xc000592840) 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery.1/pkg/apis/meta/v1/deepcopy.go:33 +0xe8 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*TableRow).DeepCopyInto(...) 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1068 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*Table).DeepCopyInto(0xc005435830, 0xc0054359e0) 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1001 +0x27c 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*Table).DeepCopy(...) 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1013 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery/pkg/apis/meta/v1.(*Table).DeepCopyObject(0xc005435830) 2022-01-27T15:02:37.171735296Z k8s.io/apimachinery.1/pkg/apis/meta/v1/zz_generated.deepcopy.go:1019 +0x45 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/audit.copyWithoutManagedFields({0x46faf80, 0xc005435830}) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/audit/request.go:291 +0xb2 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/audit.LogResponseObject({0x4728f38, 0xc003981770}, {0x46faf80, 0xc005435830}, {{0xc005374d41, 0x0}, {0xc005374d3c, 0x0}}, {0x47164d8, 0xc0073c2980}) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/audit/request.go:210 +0x2da 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated({0x47164d8, 0xc0073c2980}, {0x4713c88, 0xc001efb520}, {{0xc005374d41, 0x40ff9db}, {0xc005374d3c, 0x40ff9db}}, {0x4716778, 0xc004d5da00}, ...) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/handlers/responsewriters/writers.go:271 +0x3ff 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject({0x4728f38, 0xc003981980}, 0xc001efb520, 0x4, _, {_, _}, _, {0x0, {0x0, ...}, ...}, ...) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/handlers/response.go:137 +0x565 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/handlers.getResourceHandler.func1({0x4716778, 0xc004d5da00}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/handlers/get.go:79 +0x7fc 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints.restfulGetResource.func1(0xc0039818c0, 0xc000c1bc00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/installer.go:1221 +0x5b 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc0039818c0, 0xc000c1bc00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/metrics/metrics.go:509 +0x223 2022-01-27T15:02:37.171735296Z github.com/emicklei/go-restful.(*Container).dispatch(0xc0005b5560, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z github.com/emicklei/go-restful.5+incompatible/container.go:288 +0x9d1 2022-01-27T15:02:37.171735296Z github.com/emicklei/go-restful.(*Container).Dispatch(...) 2022-01-27T15:02:37.171735296Z github.com/emicklei/go-restful.5+incompatible/container.go:199 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4124941, 0x203001}, 0xc0005b5560, 0xc000755f80}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/handler.go:146 +0x5bb 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc003423040, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:254 +0x385 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:234 +0x70 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x411eed5, 0x203001}, 0xc0036577a0, 0xc000768380}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/handler.go:154 +0x75c 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc002172b00, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:254 +0x385 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:234 +0x70 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x411efed, 0x203001}, 0xc00200eb40, 0xc00036d0a0}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/handler.go:154 +0x75c 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc004724980, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:254 +0x385 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:234 +0x70 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4127dd7, 0x203001}, 0xc0018830e0, 0xc000670c40}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/handler.go:154 +0x75c 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc000890000, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:254 +0x385 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:234 +0x70 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4127f2c, 0xc00312a4f0}, 0xc0005b4d80, 0xc000349960}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/handler.go:154 +0x75c 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc005d5ed40, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:254 +0x385 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc00534a300, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/mux/pathrecorder.go:234 +0x70 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x4107c93, 0x469a9b8}, 0xc0005b53b0, 0xc000869f80}, {0x4716778, 0xc004d5d880}, 0xc0073b5c00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/handler.go:154 +0x75c 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithWebhookDuration.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filters/webhook_duration.go:31 +0x24e 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc007356ad0, {0x4716778, 0xc004d5d880}, 0x1c903ea) 2022-01-27T15:02:37.171735296Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc000597b80, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filters/authorization.go:64 +0x425 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0x64d6108, {0x4716778, 0xc004d5d880}, 0x0) 2022-01-27T15:02:37.171735296Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0x6372760, {0x4716778, 0xc004d5d880}, 0x3) 2022-01-27T15:02:37.171735296Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/server/filters/maxinflight.go:187 +0x29d 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc007356ef8, {0x4716778, 0xc004d5d880}, 0x1c903ea) 2022-01-27T15:02:37.171735296Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0xc0073c1200, {0x4716778, 0xc004d5d880}, 0xc0073c0fc0) 2022-01-27T15:02:37.171735296Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171735296Z k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171735296Z k8s.io/apiserver.1/pkg/endpoints/filters/impersonation.go:50 +0x21c 2022-01-27T15:02:37.171735296Z net/http.HandlerFunc.ServeHTTP(0x64d6108, {2022-01-27T15:02:37.171824805Z 0x4716778, 0xc004d5d880}, 0xa) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc0073577b8, {0x4716778, 0xc004d5d880}, 0x1c903ea) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d880}, 0xc0073b5b00) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4729de0, {0x4716778, 0xc004d5d880}, 0xc000881b80) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/audit.go:117 +0x713 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x64d6108, {0x4716778, 0xc004d5d820}, 0xa) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:80 +0x178 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc003981560, {0x4716778, 0xc004d5d820}, 0x1c903ea) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:104 +0x1a5 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4716778, 0xc004d5d820}, 0x42b2e88) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x4716778, 0xc004d5d820}, 0xc0073b5a00) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/authentication.go:80 +0x839 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f00, {0x4716778, 0xc004d5d820}, 0x469a9b8) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x4716778, 0xc004d5d820}, 0xc0073b5800) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filterlatency/filterlatency.go:89 +0x46b 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc0045b7e00, {0x4716778, 0xc004d5d820}, 0x19f0912) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1() 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/filters/timeout.go:110 +0x70 2022-01-27T15:02:37.171824805Z created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/filters/timeout.go:96 +0x1bd 2022-01-27T15:02:37.171824805Z 2022-01-27T15:02:37.171824805Z goroutine 3447 [running]: 2022-01-27T15:02:37.171824805Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3b8f000, 0xc0020d9e00}) 2022-01-27T15:02:37.171824805Z k8s.io/apimachinery.1/pkg/util/runtime/runtime.go:74 +0x7d 2022-01-27T15:02:37.171824805Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc00068c3e8, 0x1, 0x29396e6}) 2022-01-27T15:02:37.171824805Z k8s.io/apimachinery.1/pkg/util/runtime/runtime.go:48 +0x75 2022-01-27T15:02:37.171824805Z panic({0x3b8f000, 0xc0020d9e00}) 2022-01-27T15:02:37.171824805Z runtime/panic.go:1038 +0x215 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0016c75d8, {0x4708a98, 0xc000012e88}, 0xdf8475800) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/filters/timeout.go:116 +0x337 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x4708a98, 0xc000012e88}, 0xc0073b5700) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/request_deadline.go:100 +0x494 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc0073b5700, {0x4708a98, 0xc000012e88}, 0x100000000000000) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x4708a98, 0xc000012e88}, 0xc0073b5700) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/filters/waitgroup.go:77 +0x766 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0x88) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithAuditAnnotations.func1({0x4708a98, 0xc000012e88}, 0xc0073b5600) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/audit_annotations.go:36 +0x230 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0x469a9b8) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x4708a98, 0xc000012e88}, 0xc0073b5500) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/warning.go:35 +0x2bb 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3fa4aa0, {0x4708a98, 0xc000012e88}, 0xd) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x4708a98, 0xc000012e88}, 0x3) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/cachecontrol.go:31 +0x126 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x203001, {0x4708a98, 0xc000012e88}, 0x30) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x4708a98, 0xc000012e88}, 0xc0073b5500) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/httplog/httplog.go:108 +0xaf 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0x469a9b8) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x4708a98, 0xc000012e88}, 0xc0073b5400) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/requestinfo.go:39 +0x316 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x4728f38, {0x4708a98, 0xc000012e88}, 0xc0050e78f0) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x4708a98, 0xc000012e88}, 0xc0073b5300) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/request_received_time.go:38 +0x27e 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x1476d22, {0x4708a98, 0xc000012e88}, 0x0) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x4708a98, 0xc000012e88}, 0xc0073b5300) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x2c2 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3c91dc0, {0x4708a98, 0xc000012e88}, 0x100000000000008) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x4708a98, 0xc000012e88}, 0xc005359e40) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/filters/wrap.go:74 +0xba 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3eb80c0, {0x4708a98, 0xc000012e88}, 0x8) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x4708a98, 0xc000012e88}, 0xc00893e100) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/endpoints/filters/with_auditid.go:66 +0x40d 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0x3eb80c0, {0x4708a98, 0xc000012e88}, 0xd) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z github.com/openshift/library-go/pkg/apiserver/apiserverconfig.WithCacheControl.func1({0x4708a98, 0xc000012e88}, 0xc00893e100) 2022-01-27T15:02:37.171824805Z github.com/openshift/library-go.0-20211216141749-1816407208cf/pkg/apiserver/apiserverconfig/cachecontrol.go:33 +0x1bd 2022-01-27T15:02:37.171824805Z net/http.HandlerFunc.ServeHTTP(0xc001adbca0, {0x4708a98, 0xc000012e88}, 0x24) 2022-01-27T15:02:37.171824805Z net/http/server.go:2046 +0x2f 2022-01-27T15:02:37.171824805Z k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0, {0x4708a98, 0xc000012e88}, 0x4728ec8) 2022-01-27T15:02:37.171824805Z k8s.io/apiserver.1/pkg/server/handler.go:189 +0x2b 2022-01-27T15:02:37.171824805Z net/http.serverHandler.ServeHTTP({0x0}, {0x4708a98, 0xc000012e88}, 0xc00893e100) 2022-01-27T15:02:37.171824805Z net/http/server.go:2878 +0x43b 2022-01-27T15:02:37.171824805Z net/http.initALPNRequest.ServeHTTP({{0x4728f38, 0xc00339bcb0}, 0xc003dc9880, {0xc004a321c0}}, {0x4708a98, 0xc000012e88}, 0xc00893e100) 2022-01-27T15:02:37.171824805Z net/http/server.go:3479 +0x245 2022-01-27T15:02:37.171824805Z golang.org/x/net/http2.(*serverConn).runHandler(0x4700ef8, 0x64d6108, 0x0, 0x0) 2022-01-27T15:02:37.171824805Z golang.org/x/net.0-20211209124913-491a49abca63/http2/server.go:2184 +0x78 2022-01-27T15:02:37.171824805Z created by golang.org/x/net/http2.(*serverConn).processHeaders 2022-01-27T15:02:37.171824805Z golang.org/x/net.0-20211209124913-491a49abca63/http2/server.go:1914 +0x590 2022-01-27T15:02:37.171824805Z E0127 15:02:37.171716 1 wrap.go:58] "apiserver panic'd" method="GET" URI="/apis/project.openshift.io/v1/projects/default" audit-ID="fbb1a4ce-23f9-4a5e-89ba-5e7cc1420a48" ... --- Additional comment from Abu Kashem on 2022-01-27 21:56:02 UTC --- xxia, can you attach the log file with this error? --- Additional comment from Abu Kashem on 2022-01-27 21:56:56 UTC --- also, do you see similar error in kube-apiserver? --- Additional comment from Xingxing Xia on 2022-01-28 02:06:47 UTC --- Attaching the log file. I ran below OAS resources: oc get project, oc new-project test-proj, oc delete project test-proj, oc get route, oc get build, only 'oc get project' triggered the panic. I ran KAS resources like oc get pod, oc get ns, no such issue. I ran OauthAS resources like oc get user, oc get identity, no such issue. Then I tested WriteRequestBodies, 'oc get project' does not have such issue. Then I tested latest 4.9 AllRequestBodies, no such issue, so it is a 4.10 regression, adding the Keywords. --- Additional comment from Abu Kashem on 2022-01-28 19:36:59 UTC --- xxia Thanks for the information, I tracked it down, I think openshift/openshift-apiserver/pull/272 should fix the issue. Please verify when you have the next build
> oc get projects will panic if audit policy is set to AllRequestBodies It should be a blocker+ for 4.10
xxia, can you verify that every openshift API works as expected (no panic)
xxia, can you confirm that `oc get scc` also causes a panic with the current build?
oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.ci-2022-02-01-092557 True False 16m Cluster version is 4.10.0-0.ci-2022-02-01-092557 [rgangwar@preserver-workloadrhel-1 ~]$ oc get pod -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-rgangwar-01de2-dt8bf-master-0.c.openshift-qe.internal 5/5 Running 0 21m apiserver=true,app=openshift-kube-apiserver,revision=5 kube-apiserver-rgangwar-01de2-dt8bf-master-1.c.openshift-qe.internal 5/5 Running 0 18m apiserver=true,app=openshift-kube-apiserver,revision=5 kube-apiserver-rgangwar-01de2-dt8bf-master-2.c.openshift-qe.internal 5/5 Running 0 20m apiserver=true,app=openshift-kube-apiserver,revision=5 [rgangwar@preserver-workloadrhel-1 ~]$ oc edit apiserver/cluster apiserver.config.openshift.io/cluster edited oc get pod -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-rgangwar-01de2-dt8bf-master-0.c.openshift-qe.internal 5/5 Running 0 3m42s apiserver=true,app=openshift-kube-apiserver,revision=6 kube-apiserver-rgangwar-01de2-dt8bf-master-1.c.openshift-qe.internal 5/5 Running 0 5m56s apiserver=true,app=openshift-kube-apiserver,revision=6 kube-apiserver-rgangwar-01de2-dt8bf-master-2.c.openshift-qe.internal 5/5 Running 0 86s apiserver=true,app=openshift-kube-apiserver,revision=6 oc get apiserver/cluster -o yaml apiVersion: config.openshift.io/v1 kind: APIServer metadata: annotations: include.release.openshift.io/ibm-cloud-managed: "true" include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" oauth-apiserver.openshift.io/secure-token-storage: "true" release.openshift.io/create-only: "true" creationTimestamp: "2022-02-01T11:52:32Z" generation: 2 name: cluster ownerReferences: - apiVersion: config.openshift.io/v1 kind: ClusterVersion name: version uid: 0dde0018-5415-4e3b-8c61-5a3371429bff resourceVersion: "33779" uid: 14d90180-42fd-420d-bcf4-b3a22eb74ac1 spec: audit: profile: AllRequestBodies oc get project default NAME DISPLAY NAME STATUS default Active [rgangwar@preserver-workloadrhel-1 ~]$ oc get projects NAME DISPLAY NAME STATUS default Active kube-node-lease Active
moving it back to assigned since we found two more cases of panic with SCC and Templates - once the master PR https://github.com/openshift/openshift-apiserver/pull/275 merges we will need to back port to 4.10
oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-02-04-015640 True False 34m Cluster version is 4.10.0-0.nightly-2022-02-04-015640 rahulgangwar@rgangwar-mac ~ % oc get pod -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-rgangwar-04de0-t8zdz-master-0.c.openshift-qe.internal 5/5 Running 0 38m apiserver=true,app=openshift-kube-apiserver,revision=8 kube-apiserver-rgangwar-04de0-t8zdz-master-1.c.openshift-qe.internal 5/5 Running 0 39m apiserver=true,app=openshift-kube-apiserver,revision=8 kube-apiserver-rgangwar-04de0-t8zdz-master-2.c.openshift-qe.internal 5/5 Running 0 36m apiserver=true,app=openshift-kube-apiserver,revision=8 rahulgangwar@rgangwar-mac ~ % oc edit apiserver/cluster apiserver.config.openshift.io/cluster edited oc get pod -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-rgangwar-04de0-t8zdz-master-0.c.openshift-qe.internal 5/5 Running 0 10m apiserver=true,app=openshift-kube-apiserver,revision=9 kube-apiserver-rgangwar-04de0-t8zdz-master-1.c.openshift-qe.internal 5/5 Running 0 12m apiserver=true,app=openshift-kube-apiserver,revision=9 kube-apiserver-rgangwar-04de0-t8zdz-master-2.c.openshift-qe.internal 5/5 Running 0 15m apiserver=true,app=openshift-kube-apiserver,revision=9 oc get project default NAME DISPLAY NAME STATUS default Active oc get templates -n openshift|head NAME DESCRIPTION PARAMETERS OBJECTS 3scale-gateway 3scale's APIcast is an NGINX based API gateway used to integrate your interna... 17 (8 blank) 3 amq63-basic Application template for JBoss A-MQ brokers. These can be deployed as standal... 11 (4 blank) 6 amq63-persistent An example JBoss A-MQ application. For more information about using this temp... 13 (4 blank) 8 amq63-persistent-ssl An example JBoss A-MQ application. For more information about using this temp... 18 (6 blank) 12 amq63-ssl An example JBoss A-MQ application. For more information about using this temp... 16 (6 blank) 10 apicurito Design beautiful, functional APIs with zero coding, using a visual designer f... 7 (1 blank) 7 cache-service Red Hat Data Grid is an in-memory, distributed key/value store. 8 (1 blank) 4 cakephp-mysql-example An example CakePHP application with a MySQL database. For more information ab... 21 (4 blank) 8 cakephp-mysql-persistent An example CakePHP application with a MySQL database. For more information ab... 22 (4 blank) 9 cat scc.yaml kind: SecurityContextConstraints apiVersion: v1 metadata: name: scc-admin allowPrivilegedContainer: true runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny fsGroup: type: RunAsAny supplementalGroups: type: RunAsAny users: - my-admin-user groups: - my-admin-group oc create -f scc.yaml W0204 13:31:44.213822 72388 shim_kubectl.go:55] Using non-groupfied API resources is deprecated and will be removed in a future release, update apiVersion to "security.openshift.io/v1" for your resource securitycontextconstraints.security.openshift.io/scc-admin created oc get scc scc-admin NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP PRIORITY READONLYROOTFS VOLUMES scc-admin true <no value> RunAsAny RunAsAny RunAsAny RunAsAny <no value> false ["awsElasticBlockStore","azureDisk","azureFile","cephFS","cinder","configMap","csi","downwardAPI","emptyDir","ephemeral","fc","flexVolume","flocker","gcePersistentDisk","gitRepo","glusterfs","iscsi","nfs","persistentVolumeClaim","photonPersistentDisk","portworxVolume","projected","quobyte","rbd","scaleIO","secret","storageOS","vsphere"]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056