Bug 2049155

Summary:	'oc get project' caused 'Observed a panic: cannot deep copy core.NamespacePhase' when AllRequestBodies is used
Product:	OpenShift Container Platform	Reporter:	Sergiusz Urbaniak <surbania>
Component:	oauth-apiserver	Assignee:	Standa Laznicka <slaznick>
Status:	CLOSED NOTABUG	QA Contact:	Xingxing Xia <xxia>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.10	CC:	akashem, aos-bugs, mfojtik, surbania, ytripath
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2049156 (view as bug list)		Environment:
Last Closed:	2022-02-15 15:26:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2049156

Description Sergiusz Urbaniak 2022-02-01 16:36:10 UTC

This bug was initially created as a copy of Bug #2047335

I am copying this bug because: 



Description of problem:
'oc get project' caused 'Observed a panic: cannot deep copy core.NamespacePhase' when AllRequestBodies is used

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-26-234447

How reproducible:
Always

Steps to Reproduce:
1. oc edit apiserver cluster, change audit policy to AllRequestBodies.
2. Wait for pods to finish rotation
3. $ oc get project
Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get projects.project.openshift.io)

$ oc get project default
Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get projects.project.openshift.io default)

Actual results:
3. As above

Expected results:
3. Should succeed

Additional info:
OAS pod logs panic stack in next comment

Comment 4 Yash Tripathi 2022-02-04 07:16:21 UTC

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-02-04-050619   True        False         26m     Cluster version is 4.11.0-0.nightly-2022-02-04-050619

$ oc edit apiserver/cluster
apiserver.config.openshift.io/cluster edited

$ oc get pod -n openshift-kube-apiserver -l apiserver
NAME                                           READY   STATUS    RESTARTS   AGE
kube-apiserver-rgangwar-03de5-89cgh-master-0   5/5     Running   0          40s
kube-apiserver-rgangwar-03de5-89cgh-master-1   5/5     Running   0          6m37s
kube-apiserver-rgangwar-03de5-89cgh-master-2   5/5     Running   0          3m38s

$ oc get project default
NAME      DISPLAY NAME   STATUS
default                  Active

$ oc logs kube-apiserver-rgangwar-03de5-89cgh-master-0 -n openshift-kube-apiserver
/**snipped**/
I0204 07:01:45.074042      16 cacher.go:799] cacher (*core.Secret): 1 objects queued in incoming channel.
I0204 07:01:45.074065      16 cacher.go:799] cacher (*core.Secret): 2 objects queued in incoming channel.
I0204 07:01:45.101232      16 cacher.go:799] cacher (*core.ConfigMap): 3 objects queued in incoming channel.
I0204 07:01:45.101257      16 cacher.go:799] cacher (*core.ConfigMap): 4 objects queued in incoming channel.

checked in all OAS pods no panic found

Actual Results:
`oc get projects` should work

Expected Results:
`oc get projects` should work

Comment 5 Xingxing Xia 2022-02-07 02:05:56 UTC

(In reply to Sergiusz Urbaniak from comment #0)
> This bug was initially created as a copy of Bug #2047335

Thx for testing. But you steps only verified my original bug 2047335, did not test this bug 2049156 actually. This bug is different than original bug 2047335. Sergiusz cloned it but did not modify the description. Instead of resource "projects", you should look at the PRs to figure out the resources to test, they are easily shown in PR files.

Comment 6 Xingxing Xia 2022-02-15 15:26:26 UTC

Dealing with this clone bug as same as https://bugzilla.redhat.com/show_bug.cgi?id=2049156#c10 .