Bug 1882665 - Openshift 4.4.8 Cluster not working
Summary: Openshift 4.4.8 Cluster not working
Keywords:
Status: CLOSED DUPLICATE of bug 1881322
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 4.7.0
Assignee: Tomáš Nožička
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-25 09:55 UTC by Maria Alonso
Modified: 2023-12-15 19:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-02 11:39:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Maria Alonso 2020-09-25 09:55:58 UTC
Description of problem:

	Openshift 4.4.8 cluster
	Customer is not able to login using user/password or kubeconfig file.
	Not able to run any 'oc' command
	No actions done before the issue

Version-Release number of selected component (if applicable):
	Openshift 4.4.8

How reproducible:
	Not reproducible


Steps to Reproduce:
1.
2.
3.

Actual results:
	Cluster not working

Expected results:


Additional info:

Certificate errors in kube-apiserver pods logs, but as this is Openshift 4.4.8 not sure if we can try to recover from expired control plane certificates:

	As of OpenShift Container Platform 4.4.8, the cluster can automatically recover from expired control plane certificates. You no longer need to perform the manual steps that were required in previous versions.

	The exception is that you must manually approve the pending node-bootstrapper certificate signing requests (CSRs) to recover kubelet certificates.

	Use the following steps to approve the pending node-bootstrapper CSRs.

	[1] https://docs.openshift.com/container-platform/4.4/backup_and_restore/disaster_recovery/scenario-3-expired-certs.html

	- kube-apiserver logs
~~~
2020-09-23T12:57:53.155285979Z E0923 12:57:53.155253       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.175647487Z E0923 12:57:53.175593       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.905669674Z E0923 12:57:53.905614       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.906774158Z E0923 12:57:53.906735       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.907565713Z E0923 12:57:53.907536       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.922249329Z E0923 12:57:53.922208       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:53.999906135Z E0923 12:57:53.999851       1 reflector.go:153] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server is currently unable to handle the request (get groups.user.openshift.io)
2020-09-23T12:57:54.068986347Z E0923 12:57:54.068936       1 reflector.go:153] github.com/openshift/client-go/user/informers/externalversions/factory.go:101: Failed to list *v1.Group: the server is currently unable to handle the request (get groups.user.openshift.io)
2020-09-23T12:57:54.105312766Z E0923 12:57:54.105249       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.106455609Z E0923 12:57:54.106411       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.107250427Z E0923 12:57:54.107219       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.108351778Z E0923 12:57:54.108325       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.11954569Z E0923 12:57:54.119509       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.156031684Z E0923 12:57:54.155981       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
2020-09-23T12:57:54.156881194Z E0923 12:57:54.156810       1 authentication.go:104] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
~~~

	- openshift-apiserver logs:

		~~~
		2020-09-23T12:47:50.113248462Z E0923 12:47:50.113202       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:48:50.113216022Z E0923 12:48:50.113172       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:49:50.113262589Z E0923 12:49:50.113210       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:50:50.113215414Z E0923 12:50:50.113164       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:51:50.113225Z E0923 12:51:50.113168       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:52:10.983033661Z E0923 12:52:10.982963       1 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
		2020-09-23T12:52:50.113227562Z E0923 12:52:50.113170       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:53:50.113271818Z E0923 12:53:50.113206       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:54:50.113236994Z E0923 12:54:50.113184       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:55:50.113221113Z E0923 12:55:50.113172       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:56:50.113221736Z E0923 12:56:50.113167       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
		2020-09-23T12:57:50.113240617Z E0923 12:57:50.113184       1 configmap_cafile_content.go:243] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"

		~~~

Comment 3 Tomáš Nožička 2020-10-02 11:39:30 UTC

*** This bug has been marked as a duplicate of bug 1881322 ***


Note You need to log in before you can comment on or make changes to this bug.