Bug 1721586 - `/etc/kubernetes/ca.crt` is not managed/updated
Summary: `/etc/kubernetes/ca.crt` is not managed/updated
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.2.0
Assignee: Ryan Phillips
QA Contact: Sunil Choudhary
URL:
Whiteboard:
: 1734796 (view as bug list)
Depends On: 1733109 1733232
Blocks: 1732166
TreeView+ depends on / blocked
 
Reported: 2019-06-18 15:31 UTC by Tomáš Nožička
Modified: 2019-12-10 19:08 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1732166 (view as bug list)
Environment:
Last Closed: 2019-10-16 06:32:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:32:16 UTC
Github openshift machine-config-operator pull 965 None None None 2019-07-22 18:16:39 UTC

Description Tomáš Nožička 2019-06-18 15:31:33 UTC
Description of problem:

`/etc/kubernetes/ca.crt` is not managed, yet looking at the default content it contains:

- root-ca (10y)
- admin-kubeconfig-signer (10y)
- kubelet-signer (1d)
- kube-control-plane-signer (1y)
- kube-apiserver-to-kubelet-signer (1y)
- kubelet-bootstrap-kubeconfig-signer (10y)

If `kube-apiserver-to-kubelet-signer` is rotated, `/etc/kubernetes/ca.crt` requires update or logs won't work.

I am not sure which of these CAs in the bundle is actually used but if it is also another one then the `kube-apiserver-to-kubelet-signer`
this recovery step (from https://docs.openshift.com/container-platform/4.1/disaster_recovery/scenario-3-expired-certs.html)

`oc get configmap kube-apiserver-to-kubelet-client-ca -n openshift-kube-apiserver-operator --template='{{ index .data "ca-bundle.crt" }}' > /etc/kubernetes/ca.crt`

is probably not a great way to solve it as it wipes the other ones.

Also when the cert is rotated in normal flow (which might be about 1/2 of a year) logs will stop working.

Setting to urgent as the suspected outcome is logs stopping to work. Feel free to adjust if this proves not to be the case, but I have force rotated the `kube-apiserver-to-kubelet-signer` and logs stopped working until executing the above mentioned recovery step.

Comment 4 Ryan Phillips 2019-06-28 15:06:28 UTC
Updating this ticket... I have local code changes that I am testing. I found a couple bugs while working on this feature 1) kubelet crashing in standalone mode https://github.com/openshift/origin/pull/23276, and, subsequently, was trying out the code changes w/ the new kubelet and ran into issue 2) where RHCOS isn't booting https://bugzilla.redhat.com/show_bug.cgi?id=1724373.

Comment 7 Sunil Choudhary 2019-07-24 17:08:39 UTC
Checked on build 4.2.0-0.nightly-2019-07-24-000310. After deleting annotations as per, cert validity increased from 1 year to 10 year. @Tomas, can you please check if this is correct?

$ oc get secret kube-apiserver-to-kubelet-signer -o yaml 
apiVersion: v1
data:
[...]
kind: Secret
metadata:
  annotations:
    auth.openshift.io/certificate-issuer: kube-apiserver-to-kubelet-signer
    auth.openshift.io/certificate-not-after: "2020-07-23T09:09:05Z"
    auth.openshift.io/certificate-not-before: "2019-07-24T09:09:05Z"
  creationTimestamp: "2019-07-24T09:22:14Z"
  name: kube-apiserver-to-kubelet-signer
  namespace: openshift-kube-apiserver-operator
  resourceVersion: "254"
  selfLink: /api/v1/namespaces/openshift-kube-apiserver-operator/secrets/kube-apiserver-to-kubelet-signer
  uid: 85e31801-adf4-11e9-a73a-0282cb83bab0
type: SecretTypeTLS

After deleting annotations:

$ oc get secret kube-apiserver-to-kubelet-signer -o yaml 
apiVersion: v1
data:
[...]
kind: Secret
metadata:
  annotations:
    auth.openshift.io/certificate-issuer: openshift-kube-apiserver-operator_kube-apiserver-to-kubelet-signer@1563964343
    auth.openshift.io/certificate-not-after: "2029-07-21T10:32:23Z"
    auth.openshift.io/certificate-not-before: "2019-07-24T10:32:22Z"
  creationTimestamp: "2019-07-24T09:22:14Z"
  labels:
    auth.openshift.io/managed-certificate-type: signer
  name: kube-apiserver-to-kubelet-signer
  namespace: openshift-kube-apiserver-operator
  resourceVersion: "29119"
  selfLink: /api/v1/namespaces/openshift-kube-apiserver-operator/secrets/kube-apiserver-to-kubelet-signer
  uid: 85e31801-adf4-11e9-a73a-0282cb83bab0
type: SecretTypeTLS

Comment 8 Seth Jennings 2019-07-29 21:27:27 UTC
bump on this.  it is blocking the backport https://github.com/openshift/machine-config-operator/pull/1000 from merging.

Comment 9 Tomáš Nožička 2019-07-30 06:04:56 UTC
just to close on this, we have talked offline and 10 years was a bug, now fixed.

Comment 12 errata-xmlrpc 2019-10-16 06:32:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922

Comment 13 Jeremiah Stuever 2019-12-10 19:08:06 UTC
*** Bug 1734796 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.