Bug 1978193 - htpasswd provider for auth is not working as expected and give 401 error when user try to login
Summary: htpasswd provider for auth is not working as expected and give 401 error when...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Standa Laznicka
QA Contact: liyao
URL:
Whiteboard:
Depends On:
Blocks: 1997906
TreeView+ depends on / blocked
 
Reported: 2021-07-01 09:38 UTC by Praveen Kumar
Modified: 2021-10-18 17:37 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The CA for API server client certs was rotated early in the lifetime of a cluster which caused the authentication's operator logic to be unable to create a CSR because a previous CSR with the same name still existed. Consequence: The kube-apiserver was unable to authenticate itself to the oauth-apiserver when sending TokenReview requests, causing authentication to fail. Fix: Use generated names for creating CSRs in the authentication operator. Result: Early rotations of the CA for API server client certificates won't cause authentication failures for OpenShift users.
Clone Of:
Environment:
Last Closed: 2021-10-18 17:37:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
must gather from cluster (2.95 MB, application/x-xz)
2021-07-01 09:38 UTC, Praveen Kumar
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-authentication-operator pull 468 0 None open Bug 1978193: csr request: use generate names to prevent getting stuck waiting for a cert 2021-07-28 13:02:34 UTC
Github openshift library-go pull 1124 0 None open Bug 1978193: fix CSR requester controller possible infinite crashloop and expose cert validation errors 2021-07-02 14:29:48 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:37:50 UTC

Description Praveen Kumar 2021-07-01 09:38:06 UTC
Created attachment 1796719 [details]
must gather from cluster

Description of problem: We have a running cluster using libvirt provider and using the htpasswd provider which is configured as per document but even then
when user try to login to cluster it get following

```
$ oc login -u kubeadmin -p admin api.crc.testing:6443 
Login failed (401 Unauthorized)
Verify you have provided correct credentials
```

Steps to Reproduce:
```
$ htpasswd -c -B -b users.htpasswd developer developer
$ htpasswd -B -b users.htpasswd kubeadmin admin
$ oc create secret generic htpass-secret --from-file=htpasswd=users.htpasswd -n openshift-config
$ cat <<EOF | oc apply -f -
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: developer
    mappingMethod: claim 
    type: HTPasswd
    htpasswd:
      fileData:
        name: htpass-secret
<=== wait till auth pod reconcile with new changes ===>
$ oc get pods -A | grep auth
openshift-authentication-operator            authentication-operator-7d8d5485f9-q7ksr                1/1     Running     0          2d5h
openshift-authentication                     oauth-openshift-b6c4c787f-bt4pr                         1/1     Running     0          6m1s
openshift-oauth-apiserver                    apiserver-9dd98d568-25rh9                               1/1     Running     0          3d5h

$ oc login -u kubeadmin -p admin api.crc.testing:6443 --v=8
I0701 15:02:32.735445  262066 loader.go:372] Config loaded from file:  /home/prkumar/.crc/machines/crc/kubeconfig
I0701 15:02:32.736340  262066 round_trippers.go:432] HEAD https://api.crc.testing:6443/
I0701 15:02:32.736358  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.744224  262066 round_trippers.go:457] Response Status: 403 Forbidden in 7 milliseconds
I0701 15:02:32.744247  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.744257  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.744263  262066 round_trippers.go:463]     Audit-Id: 0b1e1480-9342-45ec-9840-dba94d47905a
I0701 15:02:32.744269  262066 round_trippers.go:463]     Cache-Control: no-cache, private
I0701 15:02:32.744275  262066 round_trippers.go:463]     Content-Type: application/json
I0701 15:02:32.744281  262066 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0701 15:02:32.744287  262066 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 0a588bff-a842-4520-ad68-3831a20e559f
I0701 15:02:32.744293  262066 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: c5216235-cd61-4af8-a1fc-07bba493758c
I0701 15:02:32.744306  262066 round_trippers.go:463]     Content-Length: 186
I0701 15:02:32.744935  262066 request_token.go:89] GSSAPI Enabled
I0701 15:02:32.744977  262066 round_trippers.go:432] GET https://api.crc.testing:6443/.well-known/oauth-authorization-server
I0701 15:02:32.744988  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.744997  262066 round_trippers.go:442]     X-Csrf-Token: 1
I0701 15:02:32.745937  262066 round_trippers.go:457] Response Status: 200 OK in 0 milliseconds
I0701 15:02:32.745959  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.745967  262066 round_trippers.go:463]     Audit-Id: 78ca09a1-0ee3-46bb-b911-e8468fdb66e2
I0701 15:02:32.745974  262066 round_trippers.go:463]     Cache-Control: no-cache, private
I0701 15:02:32.745980  262066 round_trippers.go:463]     Content-Type: application/json
I0701 15:02:32.745985  262066 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 0a588bff-a842-4520-ad68-3831a20e559f
I0701 15:02:32.745991  262066 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: c5216235-cd61-4af8-a1fc-07bba493758c
I0701 15:02:32.745997  262066 round_trippers.go:463]     Content-Length: 573
I0701 15:02:32.746002  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.797575  262066 request_token.go:470] falling back to kubeconfig CA due to possible x509 error: x509: certificate signed by unknown authority
I0701 15:02:32.797640  262066 round_trippers.go:432] GET https://oauth-openshift.apps-crc.testing/oauth/authorize?client_id=openshift-challenging-client&code_challenge=DLxWhTO5-DuYxttveVsm_2l28yerHnC8Nx5eiF9MrQA&code_challenge_method=S256&redirect_uri=https%3A%2F%2Foauth-openshift.apps-crc.testing%2Foauth%2Ftoken%2Fimplicit&response_type=code
I0701 15:02:32.797656  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.797665  262066 round_trippers.go:442]     X-Csrf-Token: 1
I0701 15:02:32.825475  262066 round_trippers.go:457] Response Status: 401 Unauthorized in 27 milliseconds
I0701 15:02:32.825504  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.825513  262066 round_trippers.go:463]     Cache-Control: no-cache, no-store, max-age=0, must-revalidate
I0701 15:02:32.825521  262066 round_trippers.go:463]     Expires: 0
I0701 15:02:32.825528  262066 round_trippers.go:463]     Referrer-Policy: strict-origin-when-cross-origin
I0701 15:02:32.825534  262066 round_trippers.go:463]     Www-Authenticate: Basic realm="openshift"
I0701 15:02:32.825541  262066 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0701 15:02:32.825547  262066 round_trippers.go:463]     X-Xss-Protection: 1; mode=block
I0701 15:02:32.825554  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.825561  262066 round_trippers.go:463]     Content-Length: 0
I0701 15:02:32.825567  262066 round_trippers.go:463]     Pragma: no-cache
I0701 15:02:32.825574  262066 round_trippers.go:463]     X-Dns-Prefetch-Control: off
I0701 15:02:32.825580  262066 round_trippers.go:463]     X-Frame-Options: DENY
I0701 15:02:32.825664  262066 round_trippers.go:432] GET https://oauth-openshift.apps-crc.testing/oauth/authorize?client_id=openshift-challenging-client&code_challenge=DLxWhTO5-DuYxttveVsm_2l28yerHnC8Nx5eiF9MrQA&code_challenge_method=S256&redirect_uri=https%3A%2F%2Foauth-openshift.apps-crc.testing%2Foauth%2Ftoken%2Fimplicit&response_type=code
I0701 15:02:32.825680  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.825708  262066 round_trippers.go:442]     Authorization: Basic <masked>
I0701 15:02:32.825718  262066 round_trippers.go:442]     X-Csrf-Token: 1
I0701 15:02:32.932411  262066 round_trippers.go:457] Response Status: 302 Found in 106 milliseconds
I0701 15:02:32.932431  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.932439  262066 round_trippers.go:463]     Cache-Control: no-cache, no-store, max-age=0, must-revalidate
I0701 15:02:32.932445  262066 round_trippers.go:463]     Cache-Control: no-cache, no-store, max-age=0, must-revalidate
I0701 15:02:32.932450  262066 round_trippers.go:463]     Expires: 0
I0701 15:02:32.932455  262066 round_trippers.go:463]     Expires: Fri, 01 Jan 1990 00:00:00 GMT
I0701 15:02:32.932460  262066 round_trippers.go:463]     X-Frame-Options: DENY
I0701 15:02:32.932465  262066 round_trippers.go:463]     X-Xss-Protection: 1; mode=block
I0701 15:02:32.932470  262066 round_trippers.go:463]     Referrer-Policy: strict-origin-when-cross-origin
I0701 15:02:32.932475  262066 round_trippers.go:463]     Set-Cookie: ssn=MTYyNTEzMTk1Mnx6c2h2TDFDc2R5dGVsLUc0dWxkLVg2V0ptQ1djOWZkQlVmVUhqa1pFZVRDV3hBRUd2eUc4OGMtbGRLZnR3bi1fejVMTzlxbWZHRjBreEZlUGhxUkZQcUJqUmNHRGFwbkRsLWFvVVUyUlV3R2lnT09fWklwV29lU0Y0XzFaVXg0Z0kzNXU0RXAycnV4MDRrNnI4azM4MFhCem84NHFHMm5lVFE9PXyl67dDb7VoYPZPnKtRXYeRVf9Ih4yEknlbhNAn5tv6cA==; Path=/; HttpOnly; Secure
I0701 15:02:32.932483  262066 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0701 15:02:32.932491  262066 round_trippers.go:463]     Location: https://oauth-openshift.apps-crc.testing/oauth/token/implicit?code=sha256~GEx_1ytPpfofkxgwGW1twJpFifRUV8MU4zJ_XlrahmY&state=
I0701 15:02:32.932498  262066 round_trippers.go:463]     Pragma: no-cache
I0701 15:02:32.932503  262066 round_trippers.go:463]     Pragma: no-cache
I0701 15:02:32.932507  262066 round_trippers.go:463]     Content-Length: 0
I0701 15:02:32.932511  262066 round_trippers.go:463]     X-Dns-Prefetch-Control: off
I0701 15:02:32.932514  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.932565  262066 round_trippers.go:432] POST https://oauth-openshift.apps-crc.testing/oauth/token
I0701 15:02:32.932573  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.932581  262066 round_trippers.go:442]     Authorization: Basic <masked>
I0701 15:02:32.932587  262066 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I0701 15:02:32.932592  262066 round_trippers.go:442]     Accept: application/json
I0701 15:02:32.958274  262066 round_trippers.go:457] Response Status: 200 OK in 25 milliseconds
I0701 15:02:32.958294  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.958309  262066 round_trippers.go:463]     X-Frame-Options: DENY
I0701 15:02:32.958315  262066 round_trippers.go:463]     X-Xss-Protection: 1; mode=block
I0701 15:02:32.958319  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.958323  262066 round_trippers.go:463]     X-Dns-Prefetch-Control: off
I0701 15:02:32.958327  262066 round_trippers.go:463]     Cache-Control: no-cache, no-store, max-age=0, must-revalidate
I0701 15:02:32.958331  262066 round_trippers.go:463]     Cache-Control: no-cache, no-store, max-age=0, must-revalidate
I0701 15:02:32.958335  262066 round_trippers.go:463]     Expires: 0
I0701 15:02:32.958339  262066 round_trippers.go:463]     Expires: Fri, 01 Jan 1990 00:00:00 GMT
I0701 15:02:32.958343  262066 round_trippers.go:463]     Pragma: no-cache
I0701 15:02:32.958346  262066 round_trippers.go:463]     Pragma: no-cache
I0701 15:02:32.958350  262066 round_trippers.go:463]     Content-Length: 131
I0701 15:02:32.958354  262066 round_trippers.go:463]     Content-Type: application/json
I0701 15:02:32.958358  262066 round_trippers.go:463]     Referrer-Policy: strict-origin-when-cross-origin
I0701 15:02:32.958362  262066 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0701 15:02:32.958486  262066 round_trippers.go:432] GET https://api.crc.testing:6443/apis/user.openshift.io/v1/users/~
I0701 15:02:32.958495  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.958501  262066 round_trippers.go:442]     Accept: application/json, */*
I0701 15:02:32.958506  262066 round_trippers.go:442]     User-Agent: oc/4.8.0 (linux/amd64) kubernetes/1077b05
I0701 15:02:32.958512  262066 round_trippers.go:442]     Authorization: Bearer <masked>
I0701 15:02:32.960317  262066 round_trippers.go:457] Response Status: 401 Unauthorized in 1 milliseconds
I0701 15:02:32.960337  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.960348  262066 round_trippers.go:463]     Content-Length: 129
I0701 15:02:32.960354  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.960359  262066 round_trippers.go:463]     Audit-Id: ace1fa38-840d-4ea4-b9bf-2137da1d04b7
I0701 15:02:32.960363  262066 round_trippers.go:463]     Cache-Control: no-cache, private
I0701 15:02:32.960367  262066 round_trippers.go:463]     Content-Type: application/json
I0701 15:02:32.960392  262066 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0701 15:02:32.961632  262066 round_trippers.go:432] GET https://api.crc.testing:6443/api/v1/namespaces/openshift/configmaps/motd
I0701 15:02:32.961661  262066 round_trippers.go:438] Request Headers:
I0701 15:02:32.961672  262066 round_trippers.go:442]     Accept: application/json, */*
I0701 15:02:32.961681  262066 round_trippers.go:442]     User-Agent: oc/4.8.0 (linux/amd64) kubernetes/1077b05
I0701 15:02:32.963502  262066 round_trippers.go:457] Response Status: 403 Forbidden in 1 milliseconds
I0701 15:02:32.963548  262066 round_trippers.go:460] Response Headers:
I0701 15:02:32.963572  262066 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0701 15:02:32.963584  262066 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 0a588bff-a842-4520-ad68-3831a20e559f
I0701 15:02:32.963594  262066 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: c5216235-cd61-4af8-a1fc-07bba493758c
I0701 15:02:32.963601  262066 round_trippers.go:463]     Content-Length: 303
I0701 15:02:32.963606  262066 round_trippers.go:463]     Date: Thu, 01 Jul 2021 09:32:32 GMT
I0701 15:02:32.963614  262066 round_trippers.go:463]     Audit-Id: 980b69a2-4ca9-4f10-b38c-a87646a49c14
I0701 15:02:32.963622  262066 round_trippers.go:463]     Cache-Control: no-cache, private
I0701 15:02:32.963630  262066 round_trippers.go:463]     Content-Type: application/json
I0701 15:02:32.963706  262066 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
Login failed (401 Unauthorized)
Verify you have provided correct credentials.
```

Actual results:
Failed to login for user.

Expected results:
Should successful

Additional info:
```
ClusterID: 0b43e61c-afe8-4255-925c-541faac5a8b8
ClusterVersion: Stable at "4.8.0-rc.1"
ClusterOperators:
	clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 nodes are at revision 11; 0 nodes have achieved new revision 12
	clusteroperator/cloud-credential is missing
	clusteroperator/cluster-autoscaler is missing
	clusteroperator/insights is missing
	clusteroperator/kube-storage-version-migrator is missing
```

Attached must-gather logs

Comment 1 Standa Laznicka 2021-07-01 11:18:08 UTC
Is the default kubeadmin user still present on the cluster?

Comment 2 Praveen Kumar 2021-07-01 11:24:47 UTC
@Standa Laznicka No we remove the default kubeadmin user as per https://docs.openshift.com/container-platform/4.7/authentication/remove-kubeadmin.html document after having the htpasswd configured and providing one the user cluster-admin role.

Comment 3 Praveen Kumar 2021-07-02 05:47:44 UTC
Bit more debugging and looks like issue with valid cert for authenticator . I think we can able reproduce is with any cluster where we can force cert-rotation and wait till all csr are approved.

As part of CRC we force a cert rotation to have 30 days validity and during this process we can have new csr for `node:bootstrapper` and for `node:<node-name>` but we don't have updated csr for `openshift-authentication-operator:authentication-operator` and that is the reason the auth operator pod logs says that csr exist but not a valid one.


```
$ oc get csr
NAME                                       AGE    SIGNERNAME                                    REQUESTOR                                                                         CONDITION
csr-5zmq5                                  26h    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         Approved,Issued
csr-6kptk                                  2d3h   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         Approved,Issued
csr-8f94s                                  26h    kubernetes.io/kubelet-serving                 system:node:crc-rb86w-master-0                                                    Approved,Issued
csr-xmmvg                                  2d3h   kubernetes.io/kubelet-serving                 system:node:crc-rb86w-master-0                                                    Approved,Issued
system:openshift:openshift-authenticator   2d3h   kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   Approved,Issued  


$ oc delete csr system:openshift:openshift-authenticator
certificatesigningrequest.certificates.k8s.io "system:openshift:openshift-authenticator" deleted

$ oc logs authentication-operator-7d8d5485f9-fp4rr -n openshift-authentication-operator
[...]
E0702 05:38:23.236664       1 base_controller.go:264] "OpenShiftAuthenticatorCertRequester" controller failed to sync "csr-8f94s", err: certificatesigningrequests.certificates.k8s.io "system:openshift:openshift-authenticator" already exists
I0702 05:38:24.657403       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"5b8aa10a-b814-4a64-bbc3-11cfe5e25458", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for OpenShiftAuthenticatorCertRequester is found. Bootstrap is required



[prkumar@prkumar-test snc]$ oc get csr
NAME                                       AGE   SIGNERNAME                                    REQUESTOR                                                                         CONDITION                                            
csr-76b68                                  28m   kubernetes.io/kubelet-serving                 system:node:crc-n9gwv-master-0                                                    Approved,Issued                                      
csr-9df4b                                  30m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         Approved,Issued                                      
csr-lxpg7                                  2d    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         Approved,Issued                                      
csr-m9988                                  2d    kubernetes.io/kubelet-serving                 system:node:crc-n9gwv-master-0                                                    Approved,Issued                                      
system:openshift:openshift-authenticator   4s    kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   Approved,Issued      
```

Comment 4 Standa Laznicka 2021-07-02 11:58:12 UTC
I see, this is indeed going to be an issue with the forced cert-rotation combined with a CSR that's already existing because the rotation happened earlier than the CSR got autoremoved.

Comment 7 Sergiusz Urbaniak 2021-08-17 09:24:10 UTC
sprint review: @QA: please submit sprint review status

Comment 8 liyao 2021-08-20 02:19:22 UTC
Tested in fresh cluster 4.9.0-0.nightly-2021-08-18-144658 and 4.8.0-0.nightly-2021-08-18-161850
1. delete secret openshift-authenticator-certs both in 4.8 env and 4.9 env
$ oc delete secret -n openshift-oauth-apiserver openshift-authenticator-certs
secret "openshift-authenticator-certs" deleted

2. in 4.9 env, check whether new created csr is using generateName with random suffix rather than previous fixed name 'system:openshift:openshift-authenticator'  
$ oc get csr
NAME                                             AGE   SIGNERNAME                            REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
system:openshift:openshift-authenticator-cckdw   16s   kubernetes.io/kube-apiserver-client   system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued

3. in 4.9 env, check labels 'authentication.openshift.io/csr: openshift-authenticator' is added to the new csr
$ oc get csr system:openshift:openshift-authenticator-cckdw -o yaml | grep -A5 -B5 'labels'
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  creationTimestamp: "2021-08-19T08:17:22Z"
  generateName: system:openshift:openshift-authenticator-
  labels:
    authentication.openshift.io/csr: openshift-authenticator
  name: system:openshift:openshift-authenticator-cckdw
  resourceVersion: "155347"
  uid: 91c634c2-a0b0-4247-99d4-c64141bd2616
spec:

4. delete secret openshift-authenticator-certs again both in 4.8 env and 4.9 env
$ oc delete secret -n openshift-oauth-apiserver openshift-authenticator-certs
secret "openshift-authenticator-certs" deleted

5. in 4.9 env, check whether new created csr is using different random suffix from the previous one
$ oc get csr
NAME                                             AGE   SIGNERNAME                            REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
system:openshift:openshift-authenticator-7c5zp   3s    kubernetes.io/kube-apiserver-client   system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
system:openshift:openshift-authenticator-cckdw   49s   kubernetes.io/kube-apiserver-client   system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued


6. check whether there are errors in authentication-operator pod logs
$ oc logs <pod-name> -n openshift-authentication-operator
# in 4.9 env, no error appear, which is expected with the fix
# in 4.8 env, there are below errors constantly output in the authentication-operator pod logs as Comment 3, which means the bug is reproduced
/****snipped****/
E0820 02:06:48.705837       1 base_controller.go:266] OpenShiftAuthenticatorCertRequester reconciliation failed: certificatesigningrequests.certificates.k8s.io "system:openshift:openshift-authenticator" already exists
E0820 02:06:49.768551       1 base_controller.go:264] "OpenShiftAuthenticatorCertRequester" controller failed to sync "csr-j5xjr", err: certificatesigningrequests.certificates.k8s.io "system:openshift:openshift-authenticator" already exists

Comment 11 errata-xmlrpc 2021-10-18 17:37:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.