Bug 1811061 - KCM and KS do not live reload client certificates
Summary: KCM and KS do not live reload client certificates
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Tomáš Nožička
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks: 1810008 1811062
TreeView+ depends on / blocked
 
Reported: 2020-03-06 14:23 UTC by Tomáš Nožička
Modified: 2020-07-13 17:19 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1811062 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:18:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24646 0 None closed Bug 1811061: UPSTREAM: 79083: Add ability to reload client certificates from disk 2020-11-30 19:47:34 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:19:10 UTC

Description Tomáš Nožička 2020-03-06 14:23:14 UTC
We need KCM and KS to be able to reload certificates so it can recover when cert regeneration controller fixes the cert in the API and certsyncer downloads them.

Comment 3 zhou ying 2020-03-10 10:20:17 UTC
Checked with payload: 4.5.0-0.nightly-2020-03-09-212428 , set certificate retation to 15mins , when cert updated, the kube-controller-manager pod not rollout , but the kube-scheduler pod do. 

[root@dhcp-140-138 roottest]# oc get po -n openshift-kube-controller-manager
NAME                                                                 READY   STATUS      RESTARTS   AGE
installer-12-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          45m
installer-13-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          36m
installer-13-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          40m
installer-13-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          41m
kube-controller-manager-ip-10-0-143-172.us-east-2.compute.internal   4/4     Running     2          36m
kube-controller-manager-ip-10-0-158-147.us-east-2.compute.internal   4/4     Running     0          38m
kube-controller-manager-ip-10-0-167-79.us-east-2.compute.internal    4/4     Running     0          41m


[root@dhcp-140-138 roottest]# oc get po -n openshift-kube-scheduler
NAME                                                                  READY   STATUS      RESTARTS   AGE
installer-120-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          43m
installer-120-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          37m
installer-120-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          41m
installer-121-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          32m
installer-121-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          35m
installer-121-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          33m
installer-122-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          27m
installer-122-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          25m
installer-122-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          28m
installer-123-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          19m
installer-123-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          17m
installer-123-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          20m
installer-124-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          11m
installer-124-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          10m
installer-124-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          13m
installer-125-ip-10-0-143-172.us-east-2.compute.internal              0/1     Completed   0          4m36s
installer-125-ip-10-0-158-147.us-east-2.compute.internal              0/1     Completed   0          2m27s
installer-125-ip-10-0-167-79.us-east-2.compute.internal               0/1     Completed   0          5m44s
openshift-kube-scheduler-ip-10-0-143-172.us-east-2.compute.internal   1/1     Running     0          4m31s
openshift-kube-scheduler-ip-10-0-158-147.us-east-2.compute.internal   1/1     Running     0          2m22s
openshift-kube-scheduler-ip-10-0-167-79.us-east-2.compute.internal    1/1     Running     0          5m40s

Comment 5 zhou ying 2020-03-17 05:11:26 UTC
Confirmed with payload: 4.5.0-0.nightly-2020-03-15-152626 , the issue has fixed: when cert updated, the kube-controller-manager and kube-scheduler do not rollout. 

[root@dhcp-140-138 ~]# openssl s_client -connect api.yinzhou-0316.qe.devcluster.openshift.com:6443 |openssl x509 -noout -dates
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify return:1
depth=0 CN = api.yinzhou-0316.qe.devcluster.openshift.com
verify return:1
notBefore=Mar 17 04:52:46 2020 GMT
notAfter=Mar 17 05:07:47 2020 GMT
^C
[root@dhcp-140-138 ~]# openssl s_client -connect api.yinzhou-0316.qe.devcluster.openshift.com:6443 |openssl x509 -noout -dates
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify return:1
depth=0 CN = api.yinzhou-0316.qe.devcluster.openshift.com
verify return:1
notBefore=Mar 17 05:00:21 2020 GMT
notAfter=Mar 17 05:15:22 2020 GMT
^C
[root@dhcp-140-138 ~]# oc get po -n openshift-kube-controller-manager
NAME                                                                 READY   STATUS      RESTARTS   AGE
kube-controller-manager-ip-10-0-128-13.us-east-2.compute.internal    4/4     Running     7          51m
kube-controller-manager-ip-10-0-151-31.us-east-2.compute.internal    4/4     Running     5          50m
kube-controller-manager-ip-10-0-160-206.us-east-2.compute.internal   4/4     Running     4          49m

[root@dhcp-140-138 ~]# oc get po -n openshift-kube-scheduler
NAME                                                                  READY   STATUS      RESTARTS   AGE
openshift-kube-scheduler-ip-10-0-128-13.us-east-2.compute.internal    2/2     Running     3          47m
openshift-kube-scheduler-ip-10-0-151-31.us-east-2.compute.internal    2/2     Running     2          48m
openshift-kube-scheduler-ip-10-0-160-206.us-east-2.compute.internal   2/2     Running     2          46m

Comment 7 errata-xmlrpc 2020-07-13 17:18:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.