Bug 1794309

*** Bug 1793209 has been marked as a duplicate of this bug. ***

Hi Xingxing Xia,
Can you share some details on how you are going to verify this bug?

Thanks!

Hi xxia,

We found an issue with CVO - it uses a protobuf client for `SecurityContextConstraints`. So we had to fix this issue in order for us to move the default SCC to cvo manifests in 'cluster-kube-apiserver-operator'. 

The CVO fix has the following BZ and PR
https://bugzilla.redhat.com/show_bug.cgi?id=1793209
https://github.com/openshift/cluster-version-operator/pull/308

And the actual move of the default SCC is here -
Thi BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1794309
https://github.com/openshift/cluster-kube-apiserver-operator/pull/725


We are also back porting this fix to 4.3, for this we have the following PRs cherry-picked for 4.3. We cloned the BZs above as well. 
https://github.com/openshift/cluster-kube-apiserver-operator/pull/728
https://github.com/openshift/cluster-version-operator/pull/310

Once you validate that default SCC is being created by cvo when it deploys cluster-kube-apiserver-operator we want to disable default SCC bootstrap in openshift-apiserver. I have a PR open for that but it's in hold right now, waiting for qe to approve this this BZ first.
https://github.com/openshift/openshift-apiserver/pull/67

Now, in terms of testing this BZ, in addition to doing scc related regression tests, I would recommend that we need to validate that the default SCC are being created by CVO not openshift-apiserver.
- If you check the api audit log the create issued by openshift-apiserver for the default SCC shuold have a response of `AlreadyExists`
- the api audit log will not have any create issued by cvo. For that you might need to check the api audit log of the bootstrap kube-apiserver

Thanks Abu Kashem for the detailed info.
scc related regression tests not see issue in latest 4.4.0-0.nightly-2020-02-10-013941 daily test schedule.
Checked audit, found oas scc creations with AlreadyExists which to be solved by https://github.com/openshift/openshift-apiserver/pull/67 :
[root@xxiarebuild-zcpst-control-plane-0 log]# grep -nrE "securitycontext.*create" kube-apiserver/
kube-apiserver/audit.log:47696:{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"1ec0ee24-a657-4236-bebe-958348ad85f1","stage":"ResponseComplete","requestURI":"/apis/security.openshift.io/v1/securitycontextconstraints","verb":"create","user":{"username":"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa","uid":"a0bc3021-8481-4913-9712-a24a125c0d5c","groups":["system:serviceaccounts","system:serviceaccounts:openshift-apiserver","system:authenticated"]},"sourceIPs":["10.0.97.182"],"userAgent":"openshift-apiserver/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"securitycontextconstraints","name":"privileged","apiGroup":"security.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"AlreadyExists","code":409},"requestReceivedTimestamp":"2020-02-11T12:51:06.435715Z","stageTimestamp":"2020-02-11T12:51:06.459941Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:openshift:openshift-apiserver\" of ClusterRole \"cluster-admin\" to ServiceAccount \"openshift-apiserver-sa/openshift-apiserver\""}}
kube-apiserver/audit.log:47698:{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"c39356c7-266e-4c60-ac25-3304de314cc4","stage":"ResponseComplete","requestURI":"/apis/security.openshift.io/v1/securitycontextconstraints","verb":"create","user":{"username":"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa","uid":"a0bc3021-8481-4913-9712-a24a125c0d5c","groups":["system:serviceaccounts","system:serviceaccounts:openshift-apiserver","system:authenticated"]},"sourceIPs":["10.0.97.182"],"userAgent":"openshift-apiserver/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"securitycontextconstraints","name":"nonroot","apiGroup":"security.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"AlreadyExists","code":409},"req
...snipped...

(In reply to Abu Kashem from comment #6)
> ... validate that default SCC is being created by cvo when it deploys cluster-kube-apiserver-operator
> ... you might need to check the api audit log of the bootstrap kube-apiserver
Abu, On bootstrapper, I checked /var/log/bootstrap-control-plane/kube-apiserver.log also checked journalctl -b -f -u bootkube.service, not sure the right details.
Could you give more details about where to check "the api audit log of the bootstrap kube-apiserver", how to check "SCC is being created by cvo when it deploys cluster-kube-apiserver-operator"? Thanks

"BTW, also found recreations: "username":"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", "userAgent":"openshift-apiserver, "409", I might file another bug later too:"

Hi xxia,
this is known, I mentioned this is in my comment https://bugzilla.redhat.com/show_bug.cgi?id=1794309#c6. 

"Once you validate that default SCC is being created by cvo when it deploys cluster-kube-apiserver-operator we want to disable default SCC bootstrap in openshift-apiserver. I have a PR open for that but it's in hold right now, waiting for qe to approve this this BZ first.
https://github.com/openshift/openshift-apiserver/pull/67"

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581