Created attachment 1796719 [details] must gather from cluster Description of problem: We have a running cluster using libvirt provider and using the htpasswd provider which is configured as per document but even then when user try to login to cluster it get following ``` $ oc login -u kubeadmin -p admin api.crc.testing:6443 Login failed (401 Unauthorized) Verify you have provided correct credentials ``` Steps to Reproduce: ``` $ htpasswd -c -B -b users.htpasswd developer developer $ htpasswd -B -b users.htpasswd kubeadmin admin $ oc create secret generic htpass-secret --from-file=htpasswd=users.htpasswd -n openshift-config $ cat <<EOF | oc apply -f - apiVersion: config.openshift.io/v1 kind: OAuth metadata: name: cluster spec: identityProviders: - name: developer mappingMethod: claim type: HTPasswd htpasswd: fileData: name: htpass-secret <=== wait till auth pod reconcile with new changes ===> $ oc get pods -A | grep auth openshift-authentication-operator authentication-operator-7d8d5485f9-q7ksr 1/1 Running 0 2d5h openshift-authentication oauth-openshift-b6c4c787f-bt4pr 1/1 Running 0 6m1s openshift-oauth-apiserver apiserver-9dd98d568-25rh9 1/1 Running 0 3d5h $ oc login -u kubeadmin -p admin api.crc.testing:6443 --v=8 I0701 15:02:32.735445 262066 loader.go:372] Config loaded from file: /home/prkumar/.crc/machines/crc/kubeconfig I0701 15:02:32.736340 262066 round_trippers.go:432] HEAD https://api.crc.testing:6443/ I0701 15:02:32.736358 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.744224 262066 round_trippers.go:457] Response Status: 403 Forbidden in 7 milliseconds I0701 15:02:32.744247 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.744257 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.744263 262066 round_trippers.go:463] Audit-Id: 0b1e1480-9342-45ec-9840-dba94d47905a I0701 15:02:32.744269 262066 round_trippers.go:463] Cache-Control: no-cache, private I0701 15:02:32.744275 262066 round_trippers.go:463] Content-Type: application/json I0701 15:02:32.744281 262066 round_trippers.go:463] X-Content-Type-Options: nosniff I0701 15:02:32.744287 262066 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: 0a588bff-a842-4520-ad68-3831a20e559f I0701 15:02:32.744293 262066 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: c5216235-cd61-4af8-a1fc-07bba493758c I0701 15:02:32.744306 262066 round_trippers.go:463] Content-Length: 186 I0701 15:02:32.744935 262066 request_token.go:89] GSSAPI Enabled I0701 15:02:32.744977 262066 round_trippers.go:432] GET https://api.crc.testing:6443/.well-known/oauth-authorization-server I0701 15:02:32.744988 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.744997 262066 round_trippers.go:442] X-Csrf-Token: 1 I0701 15:02:32.745937 262066 round_trippers.go:457] Response Status: 200 OK in 0 milliseconds I0701 15:02:32.745959 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.745967 262066 round_trippers.go:463] Audit-Id: 78ca09a1-0ee3-46bb-b911-e8468fdb66e2 I0701 15:02:32.745974 262066 round_trippers.go:463] Cache-Control: no-cache, private I0701 15:02:32.745980 262066 round_trippers.go:463] Content-Type: application/json I0701 15:02:32.745985 262066 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: 0a588bff-a842-4520-ad68-3831a20e559f I0701 15:02:32.745991 262066 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: c5216235-cd61-4af8-a1fc-07bba493758c I0701 15:02:32.745997 262066 round_trippers.go:463] Content-Length: 573 I0701 15:02:32.746002 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.797575 262066 request_token.go:470] falling back to kubeconfig CA due to possible x509 error: x509: certificate signed by unknown authority I0701 15:02:32.797640 262066 round_trippers.go:432] GET https://oauth-openshift.apps-crc.testing/oauth/authorize?client_id=openshift-challenging-client&code_challenge=DLxWhTO5-DuYxttveVsm_2l28yerHnC8Nx5eiF9MrQA&code_challenge_method=S256&redirect_uri=https%3A%2F%2Foauth-openshift.apps-crc.testing%2Foauth%2Ftoken%2Fimplicit&response_type=code I0701 15:02:32.797656 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.797665 262066 round_trippers.go:442] X-Csrf-Token: 1 I0701 15:02:32.825475 262066 round_trippers.go:457] Response Status: 401 Unauthorized in 27 milliseconds I0701 15:02:32.825504 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.825513 262066 round_trippers.go:463] Cache-Control: no-cache, no-store, max-age=0, must-revalidate I0701 15:02:32.825521 262066 round_trippers.go:463] Expires: 0 I0701 15:02:32.825528 262066 round_trippers.go:463] Referrer-Policy: strict-origin-when-cross-origin I0701 15:02:32.825534 262066 round_trippers.go:463] Www-Authenticate: Basic realm="openshift" I0701 15:02:32.825541 262066 round_trippers.go:463] X-Content-Type-Options: nosniff I0701 15:02:32.825547 262066 round_trippers.go:463] X-Xss-Protection: 1; mode=block I0701 15:02:32.825554 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.825561 262066 round_trippers.go:463] Content-Length: 0 I0701 15:02:32.825567 262066 round_trippers.go:463] Pragma: no-cache I0701 15:02:32.825574 262066 round_trippers.go:463] X-Dns-Prefetch-Control: off I0701 15:02:32.825580 262066 round_trippers.go:463] X-Frame-Options: DENY I0701 15:02:32.825664 262066 round_trippers.go:432] GET https://oauth-openshift.apps-crc.testing/oauth/authorize?client_id=openshift-challenging-client&code_challenge=DLxWhTO5-DuYxttveVsm_2l28yerHnC8Nx5eiF9MrQA&code_challenge_method=S256&redirect_uri=https%3A%2F%2Foauth-openshift.apps-crc.testing%2Foauth%2Ftoken%2Fimplicit&response_type=code I0701 15:02:32.825680 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.825708 262066 round_trippers.go:442] Authorization: Basic <masked> I0701 15:02:32.825718 262066 round_trippers.go:442] X-Csrf-Token: 1 I0701 15:02:32.932411 262066 round_trippers.go:457] Response Status: 302 Found in 106 milliseconds I0701 15:02:32.932431 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.932439 262066 round_trippers.go:463] Cache-Control: no-cache, no-store, max-age=0, must-revalidate I0701 15:02:32.932445 262066 round_trippers.go:463] Cache-Control: no-cache, no-store, max-age=0, must-revalidate I0701 15:02:32.932450 262066 round_trippers.go:463] Expires: 0 I0701 15:02:32.932455 262066 round_trippers.go:463] Expires: Fri, 01 Jan 1990 00:00:00 GMT I0701 15:02:32.932460 262066 round_trippers.go:463] X-Frame-Options: DENY I0701 15:02:32.932465 262066 round_trippers.go:463] X-Xss-Protection: 1; mode=block I0701 15:02:32.932470 262066 round_trippers.go:463] Referrer-Policy: strict-origin-when-cross-origin I0701 15:02:32.932475 262066 round_trippers.go:463] Set-Cookie: ssn=MTYyNTEzMTk1Mnx6c2h2TDFDc2R5dGVsLUc0dWxkLVg2V0ptQ1djOWZkQlVmVUhqa1pFZVRDV3hBRUd2eUc4OGMtbGRLZnR3bi1fejVMTzlxbWZHRjBreEZlUGhxUkZQcUJqUmNHRGFwbkRsLWFvVVUyUlV3R2lnT09fWklwV29lU0Y0XzFaVXg0Z0kzNXU0RXAycnV4MDRrNnI4azM4MFhCem84NHFHMm5lVFE9PXyl67dDb7VoYPZPnKtRXYeRVf9Ih4yEknlbhNAn5tv6cA==; Path=/; HttpOnly; Secure I0701 15:02:32.932483 262066 round_trippers.go:463] X-Content-Type-Options: nosniff I0701 15:02:32.932491 262066 round_trippers.go:463] Location: https://oauth-openshift.apps-crc.testing/oauth/token/implicit?code=sha256~GEx_1ytPpfofkxgwGW1twJpFifRUV8MU4zJ_XlrahmY&state= I0701 15:02:32.932498 262066 round_trippers.go:463] Pragma: no-cache I0701 15:02:32.932503 262066 round_trippers.go:463] Pragma: no-cache I0701 15:02:32.932507 262066 round_trippers.go:463] Content-Length: 0 I0701 15:02:32.932511 262066 round_trippers.go:463] X-Dns-Prefetch-Control: off I0701 15:02:32.932514 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.932565 262066 round_trippers.go:432] POST https://oauth-openshift.apps-crc.testing/oauth/token I0701 15:02:32.932573 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.932581 262066 round_trippers.go:442] Authorization: Basic <masked> I0701 15:02:32.932587 262066 round_trippers.go:442] Content-Type: application/x-www-form-urlencoded I0701 15:02:32.932592 262066 round_trippers.go:442] Accept: application/json I0701 15:02:32.958274 262066 round_trippers.go:457] Response Status: 200 OK in 25 milliseconds I0701 15:02:32.958294 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.958309 262066 round_trippers.go:463] X-Frame-Options: DENY I0701 15:02:32.958315 262066 round_trippers.go:463] X-Xss-Protection: 1; mode=block I0701 15:02:32.958319 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.958323 262066 round_trippers.go:463] X-Dns-Prefetch-Control: off I0701 15:02:32.958327 262066 round_trippers.go:463] Cache-Control: no-cache, no-store, max-age=0, must-revalidate I0701 15:02:32.958331 262066 round_trippers.go:463] Cache-Control: no-cache, no-store, max-age=0, must-revalidate I0701 15:02:32.958335 262066 round_trippers.go:463] Expires: 0 I0701 15:02:32.958339 262066 round_trippers.go:463] Expires: Fri, 01 Jan 1990 00:00:00 GMT I0701 15:02:32.958343 262066 round_trippers.go:463] Pragma: no-cache I0701 15:02:32.958346 262066 round_trippers.go:463] Pragma: no-cache I0701 15:02:32.958350 262066 round_trippers.go:463] Content-Length: 131 I0701 15:02:32.958354 262066 round_trippers.go:463] Content-Type: application/json I0701 15:02:32.958358 262066 round_trippers.go:463] Referrer-Policy: strict-origin-when-cross-origin I0701 15:02:32.958362 262066 round_trippers.go:463] X-Content-Type-Options: nosniff I0701 15:02:32.958486 262066 round_trippers.go:432] GET https://api.crc.testing:6443/apis/user.openshift.io/v1/users/~ I0701 15:02:32.958495 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.958501 262066 round_trippers.go:442] Accept: application/json, */* I0701 15:02:32.958506 262066 round_trippers.go:442] User-Agent: oc/4.8.0 (linux/amd64) kubernetes/1077b05 I0701 15:02:32.958512 262066 round_trippers.go:442] Authorization: Bearer <masked> I0701 15:02:32.960317 262066 round_trippers.go:457] Response Status: 401 Unauthorized in 1 milliseconds I0701 15:02:32.960337 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.960348 262066 round_trippers.go:463] Content-Length: 129 I0701 15:02:32.960354 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.960359 262066 round_trippers.go:463] Audit-Id: ace1fa38-840d-4ea4-b9bf-2137da1d04b7 I0701 15:02:32.960363 262066 round_trippers.go:463] Cache-Control: no-cache, private I0701 15:02:32.960367 262066 round_trippers.go:463] Content-Type: application/json I0701 15:02:32.960392 262066 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401} I0701 15:02:32.961632 262066 round_trippers.go:432] GET https://api.crc.testing:6443/api/v1/namespaces/openshift/configmaps/motd I0701 15:02:32.961661 262066 round_trippers.go:438] Request Headers: I0701 15:02:32.961672 262066 round_trippers.go:442] Accept: application/json, */* I0701 15:02:32.961681 262066 round_trippers.go:442] User-Agent: oc/4.8.0 (linux/amd64) kubernetes/1077b05 I0701 15:02:32.963502 262066 round_trippers.go:457] Response Status: 403 Forbidden in 1 milliseconds I0701 15:02:32.963548 262066 round_trippers.go:460] Response Headers: I0701 15:02:32.963572 262066 round_trippers.go:463] X-Content-Type-Options: nosniff I0701 15:02:32.963584 262066 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: 0a588bff-a842-4520-ad68-3831a20e559f I0701 15:02:32.963594 262066 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: c5216235-cd61-4af8-a1fc-07bba493758c I0701 15:02:32.963601 262066 round_trippers.go:463] Content-Length: 303 I0701 15:02:32.963606 262066 round_trippers.go:463] Date: Thu, 01 Jul 2021 09:32:32 GMT I0701 15:02:32.963614 262066 round_trippers.go:463] Audit-Id: 980b69a2-4ca9-4f10-b38c-a87646a49c14 I0701 15:02:32.963622 262066 round_trippers.go:463] Cache-Control: no-cache, private I0701 15:02:32.963630 262066 round_trippers.go:463] Content-Type: application/json I0701 15:02:32.963706 262066 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403} Login failed (401 Unauthorized) Verify you have provided correct credentials. ``` Actual results: Failed to login for user. Expected results: Should successful Additional info: ``` ClusterID: 0b43e61c-afe8-4255-925c-541faac5a8b8 ClusterVersion: Stable at "4.8.0-rc.1" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 nodes are at revision 11; 0 nodes have achieved new revision 12 clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/kube-storage-version-migrator is missing ``` Attached must-gather logs
Is the default kubeadmin user still present on the cluster?
@Standa Laznicka No we remove the default kubeadmin user as per https://docs.openshift.com/container-platform/4.7/authentication/remove-kubeadmin.html document after having the htpasswd configured and providing one the user cluster-admin role.
Bit more debugging and looks like issue with valid cert for authenticator . I think we can able reproduce is with any cluster where we can force cert-rotation and wait till all csr are approved. As part of CRC we force a cert rotation to have 30 days validity and during this process we can have new csr for `node:bootstrapper` and for `node:<node-name>` but we don't have updated csr for `openshift-authentication-operator:authentication-operator` and that is the reason the auth operator pod logs says that csr exist but not a valid one. ``` $ oc get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-5zmq5 26h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-6kptk 2d3h kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-8f94s 26h kubernetes.io/kubelet-serving system:node:crc-rb86w-master-0 Approved,Issued csr-xmmvg 2d3h kubernetes.io/kubelet-serving system:node:crc-rb86w-master-0 Approved,Issued system:openshift:openshift-authenticator 2d3h kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator Approved,Issued $ oc delete csr system:openshift:openshift-authenticator certificatesigningrequest.certificates.k8s.io "system:openshift:openshift-authenticator" deleted $ oc logs authentication-operator-7d8d5485f9-fp4rr -n openshift-authentication-operator [...] E0702 05:38:23.236664 1 base_controller.go:264] "OpenShiftAuthenticatorCertRequester" controller failed to sync "csr-8f94s", err: certificatesigningrequests.certificates.k8s.io "system:openshift:openshift-authenticator" already exists I0702 05:38:24.657403 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"5b8aa10a-b814-4a64-bbc3-11cfe5e25458", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for OpenShiftAuthenticatorCertRequester is found. Bootstrap is required [prkumar@prkumar-test snc]$ oc get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-76b68 28m kubernetes.io/kubelet-serving system:node:crc-n9gwv-master-0 Approved,Issued csr-9df4b 30m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-lxpg7 2d kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-m9988 2d kubernetes.io/kubelet-serving system:node:crc-n9gwv-master-0 Approved,Issued system:openshift:openshift-authenticator 4s kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator Approved,Issued ```
I see, this is indeed going to be an issue with the forced cert-rotation combined with a CSR that's already existing because the rotation happened earlier than the CSR got autoremoved.
sprint review: @QA: please submit sprint review status
Tested in fresh cluster 4.9.0-0.nightly-2021-08-18-144658 and 4.8.0-0.nightly-2021-08-18-161850 1. delete secret openshift-authenticator-certs both in 4.8 env and 4.9 env $ oc delete secret -n openshift-oauth-apiserver openshift-authenticator-certs secret "openshift-authenticator-certs" deleted 2. in 4.9 env, check whether new created csr is using generateName with random suffix rather than previous fixed name 'system:openshift:openshift-authenticator' $ oc get csr NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION system:openshift:openshift-authenticator-cckdw 16s kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator <none> Approved,Issued 3. in 4.9 env, check labels 'authentication.openshift.io/csr: openshift-authenticator' is added to the new csr $ oc get csr system:openshift:openshift-authenticator-cckdw -o yaml | grep -A5 -B5 'labels' apiVersion: certificates.k8s.io/v1 kind: CertificateSigningRequest metadata: creationTimestamp: "2021-08-19T08:17:22Z" generateName: system:openshift:openshift-authenticator- labels: authentication.openshift.io/csr: openshift-authenticator name: system:openshift:openshift-authenticator-cckdw resourceVersion: "155347" uid: 91c634c2-a0b0-4247-99d4-c64141bd2616 spec: 4. delete secret openshift-authenticator-certs again both in 4.8 env and 4.9 env $ oc delete secret -n openshift-oauth-apiserver openshift-authenticator-certs secret "openshift-authenticator-certs" deleted 5. in 4.9 env, check whether new created csr is using different random suffix from the previous one $ oc get csr NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION system:openshift:openshift-authenticator-7c5zp 3s kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator <none> Approved,Issued system:openshift:openshift-authenticator-cckdw 49s kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator <none> Approved,Issued 6. check whether there are errors in authentication-operator pod logs $ oc logs <pod-name> -n openshift-authentication-operator # in 4.9 env, no error appear, which is expected with the fix # in 4.8 env, there are below errors constantly output in the authentication-operator pod logs as Comment 3, which means the bug is reproduced /****snipped****/ E0820 02:06:48.705837 1 base_controller.go:266] OpenShiftAuthenticatorCertRequester reconciliation failed: certificatesigningrequests.certificates.k8s.io "system:openshift:openshift-authenticator" already exists E0820 02:06:49.768551 1 base_controller.go:264] "OpenShiftAuthenticatorCertRequester" controller failed to sync "csr-j5xjr", err: certificatesigningrequests.certificates.k8s.io "system:openshift:openshift-authenticator" already exists
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759