Bug 1721586

Summary:	`/etc/kubernetes/ca.crt` is not managed/updated
Product:	OpenShift Container Platform	Reporter:	Tomáš Nožička <tnozicka>
Component:	Node	Assignee:	Ryan Phillips <rphillips>
Status:	CLOSED ERRATA	QA Contact:	Sunil Choudhary <schoudha>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.2.0	CC:	adahiya, aos-bugs, clasohm, deads, jokerman, mmccomas, rsawhill, sdodson, sjenning
Target Milestone:	---	Keywords:	NeedsTestCase
Target Release:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1732166 (view as bug list)		Environment:
Last Closed:	2019-10-16 06:32:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1733109, 1733232
Bug Blocks:	1732166

Description Tomáš Nožička 2019-06-18 15:31:33 UTC

Description of problem:

`/etc/kubernetes/ca.crt` is not managed, yet looking at the default content it contains:

- root-ca (10y)
- admin-kubeconfig-signer (10y)
- kubelet-signer (1d)
- kube-control-plane-signer (1y)
- kube-apiserver-to-kubelet-signer (1y)
- kubelet-bootstrap-kubeconfig-signer (10y)

If `kube-apiserver-to-kubelet-signer` is rotated, `/etc/kubernetes/ca.crt` requires update or logs won't work.

I am not sure which of these CAs in the bundle is actually used but if it is also another one then the `kube-apiserver-to-kubelet-signer`
this recovery step (from https://docs.openshift.com/container-platform/4.1/disaster_recovery/scenario-3-expired-certs.html)

`oc get configmap kube-apiserver-to-kubelet-client-ca -n openshift-kube-apiserver-operator --template='{{ index .data "ca-bundle.crt" }}' > /etc/kubernetes/ca.crt`

is probably not a great way to solve it as it wipes the other ones.

Also when the cert is rotated in normal flow (which might be about 1/2 of a year) logs will stop working.

Setting to urgent as the suspected outcome is logs stopping to work. Feel free to adjust if this proves not to be the case, but I have force rotated the `kube-apiserver-to-kubelet-signer` and logs stopped working until executing the above mentioned recovery step.

Comment 4 Ryan Phillips 2019-06-28 15:06:28 UTC

Updating this ticket... I have local code changes that I am testing. I found a couple bugs while working on this feature 1) kubelet crashing in standalone mode https://github.com/openshift/origin/pull/23276, and, subsequently, was trying out the code changes w/ the new kubelet and ran into issue 2) where RHCOS isn't booting https://bugzilla.redhat.com/show_bug.cgi?id=1724373.

Comment 5 Ryan Phillips 2019-07-12 21:23:44 UTC

PR: https://github.com/openshift/machine-config-operator/pull/965

Comment 7 Sunil Choudhary 2019-07-24 17:08:39 UTC

Checked on build 4.2.0-0.nightly-2019-07-24-000310. After deleting annotations as per, cert validity increased from 1 year to 10 year. @Tomas, can you please check if this is correct?

$ oc get secret kube-apiserver-to-kubelet-signer -o yaml 
apiVersion: v1
data:
[...]
kind: Secret
metadata:
  annotations:
    auth.openshift.io/certificate-issuer: kube-apiserver-to-kubelet-signer
    auth.openshift.io/certificate-not-after: "2020-07-23T09:09:05Z"
    auth.openshift.io/certificate-not-before: "2019-07-24T09:09:05Z"
  creationTimestamp: "2019-07-24T09:22:14Z"
  name: kube-apiserver-to-kubelet-signer
  namespace: openshift-kube-apiserver-operator
  resourceVersion: "254"
  selfLink: /api/v1/namespaces/openshift-kube-apiserver-operator/secrets/kube-apiserver-to-kubelet-signer
  uid: 85e31801-adf4-11e9-a73a-0282cb83bab0
type: SecretTypeTLS

After deleting annotations:

$ oc get secret kube-apiserver-to-kubelet-signer -o yaml 
apiVersion: v1
data:
[...]
kind: Secret
metadata:
  annotations:
    auth.openshift.io/certificate-issuer: openshift-kube-apiserver-operator_kube-apiserver-to-kubelet-signer@1563964343
    auth.openshift.io/certificate-not-after: "2029-07-21T10:32:23Z"
    auth.openshift.io/certificate-not-before: "2019-07-24T10:32:22Z"
  creationTimestamp: "2019-07-24T09:22:14Z"
  labels:
    auth.openshift.io/managed-certificate-type: signer
  name: kube-apiserver-to-kubelet-signer
  namespace: openshift-kube-apiserver-operator
  resourceVersion: "29119"
  selfLink: /api/v1/namespaces/openshift-kube-apiserver-operator/secrets/kube-apiserver-to-kubelet-signer
  uid: 85e31801-adf4-11e9-a73a-0282cb83bab0
type: SecretTypeTLS

Comment 8 Seth Jennings 2019-07-29 21:27:27 UTC

bump on this.  it is blocking the backport https://github.com/openshift/machine-config-operator/pull/1000 from merging.

Comment 9 Tomáš Nožička 2019-07-30 06:04:56 UTC

just to close on this, we have talked offline and 10 years was a bug, now fixed.

Comment 12 errata-xmlrpc 2019-10-16 06:32:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922

Comment 13 Jeremiah Stuever 2019-12-10 19:08:06 UTC

*** Bug 1734796 has been marked as a duplicate of this bug. ***