Bug 1882665

Description of problem:

	Openshift 4.4.8 cluster
	Customer is not able to login using user/password or kubeconfig file.
	Not able to run any 'oc' command
	No actions done before the issue

Version-Release number of selected component (if applicable):
	Openshift 4.4.8

How reproducible:
	Not reproducible


Steps to Reproduce:
1.
2.
3.

Actual results:
	Cluster not working

Expected results:


Additional info:

Certificate errors in kube-apiserver pods logs, but as this is Openshift 4.4.8 not sure if we can try to recover from expired control plane certificates:

	As of OpenShift Container Platform 4.4.8, the cluster can automatically recover from expired control plane certificates. You no longer need to perform the manual steps that were required in previous versions.

	The exception is that you must manually approve the pending node-bootstrapper certificate signing requests (CSRs) to recover kubelet certificates.

	Use the following steps to approve the pending node-bootstrapper CSRs.

	[1] https://docs.openshift.com/container-platform/4.4/backup_and_restore/disaster_recovery/scenario-3-expired-certs.html

	- kube-apiserver logs
~~~
2020-09-23T12:57:53.155285979Z E0923 12:57:53.155253       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.175647487Z E0923 12:57:53.175593       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.905669674Z E0923 12:57:53.905614       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.906774158Z E0923 12:57:53.906735       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.907565713Z E0923 12:57:53.907536       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.922249329Z E0923 12:57:53.922208       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.999906135Z E0923 12:57:53.999851       1 reflector.go:153] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server is currently unable to handle the request (get groups.user.openshift.io)
2020-09-23T12:57:54.068986347Z E0923 12:57:54.068936       1 reflector.go:153] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server is currently unable to handle the request (get groups.user.openshift.io)
2020-09-23T12:57:54.105312766Z E0923 12:57:54.105249       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.106455609Z E0923 12:57:54.106411       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.107250427Z E0923 12:57:54.107219       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.108351778Z E0923 12:57:54.108325       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.11954569Z E0923 12:57:54.119509       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.156031684Z E0923 12:57:54.155981       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.156881194Z E0923 12:57:54.156810       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
~~~

	- openshift-apiserver logs:

		~~~
		2020-09-23T12:47:50.113248462Z E0923 12:47:50.113202       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:48:50.113216022Z E0923 12:48:50.113172       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:49:50.113262589Z E0923 12:49:50.113210       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:50:50.113215414Z E0923 12:50:50.113164       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:51:50.113225Z E0923 12:51:50.113168       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:52:10.983033661Z E0923 12:52:10.982963       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
		2020-09-23T12:52:50.113227562Z E0923 12:52:50.113170       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:53:50.113271818Z E0923 12:53:50.113206       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:54:50.113236994Z E0923 12:54:50.113184       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:55:50.113221113Z E0923 12:55:50.113172       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:56:50.113221736Z E0923 12:56:50.113167       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:57:50.113240617Z E0923 12:57:50.113184       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"

		~~~