Bug 1813911 - recovery is stuck on kube-system::extension-apiserver-authentication::requestheader-client-ca-file configmap
Summary: recovery is stuck on kube-system::extension-apiserver-authentication::request...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.5.0
Assignee: Tomáš Nožička
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks: 1813914
TreeView+ depends on / blocked
 
Reported: 2020-03-16 13:21 UTC by Tomáš Nožička
Modified: 2020-07-13 17:20 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1813914 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:20:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 797 0 None closed Bug 1813911: Disable serving for cert regeneration controller to avoid wrong dependency 2020-07-22 12:41:32 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:20:41 UTC

Description Tomáš Nožička 2020-03-16 13:21:00 UTC
Server Version: 4.4.0-0.nightly-2020-03-12-152413
Kubernetes Version: v1.17.1

kube-apiserver-cert-regeneration-controller container logs:

E0316 11:09:35.761846       1 configmap_cafile_content.go:246] key failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
E0316 11:10:06.515529       1 configmap_cafile_content.go:246] kube-system/extension-apiserver-authentication failed with : missing content for CA bundle "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"


it blocks the recovery controller from starting

Comment 5 Xingxing Xia 2020-03-19 02:28:48 UTC
Checked this morning again:
[xxia@pres 2020-03-19 10:04:49 CST ~]$ oc get po -n openshift-kube-apiserver -l apiserver --show-labels
NAME                                                    READY STATUS  RESTARTS AGE   LABELS
kube-apiserver-xxia03-9pprc-m-0.c.openshift-qe.internal 4/4   Running 0        3h36m apiserver=true,app=openshift-kube-apiserver,revision=13
kube-apiserver-xxia03-9pprc-m-1.c.openshift-qe.internal 4/4   Running 0        3h38m apiserver=true,app=openshift-kube-apiserver,revision=13
kube-apiserver-xxia03-9pprc-m-2.c.openshift-qe.internal 4/4   Running 0        3h40m apiserver=true,app=openshift-kube-apiserver,revision=13

[xxia@pres 2020-03-19 10:15:35 CST my]$ cat scripts/check_secret_expiry_2.sh
oc get secret --insecure-skip-tls-verify -A -o json | jq -r '.items[] | select(.metadata.annotations."auth.openshift.io/certificate-not-after" | . != null and fromdateiso8601<='$( date --date='+24hours' +%s )') | "\(.metadata.annotations."auth.openshift.io/certificate-not-before")  \(.metadata.annotations."auth.openshift.io/certificate-not-after")  \(.metadata.namespace)\t\(.metadata.name)"'

# found expiry of below last four certs is still not recovered as of the time shown in the shell prompt
[xxia@pres 2020-03-19 10:15:35 CST my]$ scripts/check_secret_expiry_2.sh
2020-03-18T22:53:46Z  2020-03-19T10:53:47Z  openshift-config-managed    kube-controller-manager-client-cert-key
2020-03-18T22:53:46Z  2020-03-19T10:53:47Z  openshift-config-managed    kube-scheduler-client-cert-key
2020-03-18T22:53:47Z  2020-03-19T10:53:48Z  openshift-kube-apiserver-operator   aggregator-client-signer
2020-03-18T21:36:18Z  2020-03-19T04:53:49Z  openshift-kube-apiserver    aggregator-client
2020-03-18T22:53:46Z  2020-03-19T10:53:47Z  openshift-kube-apiserver    external-loadbalancer-serving-certkey
2020-03-18T22:53:47Z  2020-03-19T10:53:48Z  openshift-kube-apiserver    internal-loadbalancer-serving-certkey
2020-03-18T22:25:53Z  2020-03-19T10:25:54Z  openshift-kube-apiserver    kubelet-client
2020-03-18T10:25:51Z  2020-03-18T22:25:52Z  openshift-kube-apiserver    kubelet-client-11
2020-03-18T16:25:53Z  2020-03-19T04:25:54Z  openshift-kube-apiserver    kubelet-client-12
2020-03-18T22:25:53Z  2020-03-19T10:25:54Z  openshift-kube-apiserver    kubelet-client-13
2020-03-18T22:53:47Z  2020-03-19T10:53:48Z  openshift-kube-apiserver    localhost-serving-cert-certkey
2020-03-18T22:53:46Z  2020-03-19T10:53:47Z  openshift-kube-apiserver    service-network-serving-certkey
2020-03-18T10:25:54Z  2020-03-18T12:25:55Z  openshift-kube-controller-manager-operator  csr-signer
2020-03-18T10:25:55Z  2020-03-18T14:25:56Z  openshift-kube-controller-manager-operator  csr-signer-signer
2020-03-18T10:53:39Z  2020-03-18T22:53:40Z  openshift-kube-controller-manager   kube-controller-manager-client-cert-key
2020-03-18T10:25:50Z  2020-03-18T10:40:51Z  openshift-kube-scheduler    kube-scheduler-client-cert-key

Comment 6 Xingxing Xia 2020-03-20 10:00:48 UTC
After confirmed with Dev, destroying the cluster of comment 3 and will do another try on next working day.

Comment 7 Xingxing Xia 2020-03-23 07:06:49 UTC
Retried in 4.5.0-0.nightly-2020-03-22-224936 ipi on gcp env, this time didn't hit previous comment issue. Moving to VERIFIED. Will open separate bug once hit again.

Comment 9 errata-xmlrpc 2020-07-13 17:20:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.