Bug 1716622 - Updating the kube-apiserver certificate with a new certificate fails to reload the kube-apiserver certificate
Summary: Updating the kube-apiserver certificate with a new certificate fails to reloa...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 4.1.0
Hardware: All
OS: All
Target Milestone: ---
: 4.1.z
Assignee: Luis Sanchez
QA Contact: Xingxing Xia
Whiteboard: 4.1.2
Depends On: 1714771
Blocks: 1718956
TreeView+ depends on / blocked
Reported: 2019-06-03 19:06 UTC by Greg Blomquist
Modified: 2019-06-19 06:45 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1714771
Last Closed: 2019-06-19 06:45:34 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1382 0 None None None 2019-06-19 06:45:44 UTC

Comment 1 Greg Blomquist 2019-06-03 19:18:53 UTC

Comment 4 W. Trevor King 2019-06-05 15:06:16 UTC
Both PRs merged.

Comment 6 Xingxing Xia 2019-06-06 04:55:34 UTC
Tested 4.1.0-0.nightly-2019-06-05-233256, updating the kube-apiserver certificate with a new certificate, still fails to reload/rollout:
First, add certificate by following https://bugzilla.redhat.com/show_bug.cgi?id=1685704#c26 :
$ openssl genrsa  -out custom2.key 1024
$ openssl req -new -key custom2.key -out custom2.csr
Common Name (eg, your name or your server's hostname) []:api.xxia-test.qe.devcluster.openshift.com
$ openssl x509 -req -days 1 -in custom2.csr -signkey custom2.key -out custom2.crt
$ oc create secret tls api-certs --cert=custom2.crt --key=custom2.key -n openshift-config
$ oc edit apiserver cluster
    - names:
      - api.xxia-test.qe.devcluster.openshift.com
        name: api-certs

Then new installer-7-ip-* pods run --> kube-apiserver pods restart.

Second, update the certificate with new .crt:
$ openssl x509 -req -days 1 -in custom2.csr -signkey custom2.key -out custom2-2.crt
$ oc create secret tls api-certs --cert=custom2-2.crt --key=custom2.key -n openshift-config --dry-run -o yaml | oc apply -f -
Watch pods, no new installer-8-ip-* appear, kube-apiserver pods never restart accordingly:
$ watch oc get po -n openshift-kube-apiserver
Every 2.0s: oc get po -n openshift-kube-apiserver                                                                                   fedora29: Thu Jun  6 12:51[0/1243]

NAME                                                           READY   STATUS      RESTARTS   AGE 
installer-2-ip-10-0-133-233.us-east-2.compute.internal         0/1     Completed   0          3h15m 
installer-2-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          3h13m 
installer-2-ip-10-0-172-171.us-east-2.compute.internal         0/1     Completed   0          3h15m 
installer-3-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          3h13m 
installer-4-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          3h12m 
installer-5-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          3h11m 
installer-6-ip-10-0-133-233.us-east-2.compute.internal         0/1     Completed   0          3h7m 
installer-6-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          3h11m 
installer-6-ip-10-0-172-171.us-east-2.compute.internal         0/1     Completed   0          3h9m 
installer-7-ip-10-0-133-233.us-east-2.compute.internal         0/1     Completed   0          18m 
installer-7-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          16m 
installer-7-ip-10-0-172-171.us-east-2.compute.internal         0/1     Completed   0          19m 
kube-apiserver-ip-10-0-133-233.us-east-2.compute.internal      2/2     Running     0          17m 
kube-apiserver-ip-10-0-159-216.us-east-2.compute.internal      2/2     Running     0          16m 
kube-apiserver-ip-10-0-172-171.us-east-2.compute.internal      2/2     Running     0          19m

Comment 7 David Eads 2019-06-06 15:02:56 UTC
The API server doesn't restart when certificates change, only when configuration changes.  Check to see if you're actually serving with the new certificates.

Also, as a reminder, please include the `oc adm must-gather` report so we can avoid ping-ponging back and forth.

Comment 8 Mike Fiedler 2019-06-06 16:05:27 UTC
Accessed the cluster in question above and the rollouts were still on the same generation:

$ oc get pods -n openshift-kube-apiserver
NAME                                                           READY   STATUS      RESTARTS   AGE
installer-2-ip-10-0-133-233.us-east-2.compute.internal         0/1     Completed   0          14h
installer-2-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          14h
installer-2-ip-10-0-172-171.us-east-2.compute.internal         0/1     Completed   0          14h
installer-3-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          14h
installer-4-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          14h
installer-5-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          14h
installer-6-ip-10-0-133-233.us-east-2.compute.internal         0/1     Completed   0          14h
installer-6-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          14h
installer-6-ip-10-0-172-171.us-east-2.compute.internal         0/1     Completed   0          14h
installer-7-ip-10-0-133-233.us-east-2.compute.internal         0/1     Completed   0          11h
installer-7-ip-10-0-159-216.us-east-2.compute.internal         0/1     Completed   0          11h
installer-7-ip-10-0-172-171.us-east-2.compute.internal         0/1     Completed   0          11h
kube-apiserver-ip-10-0-133-233.us-east-2.compute.internal      2/2     Running     0          11h
kube-apiserver-ip-10-0-159-216.us-east-2.compute.internal      2/2     Running     0          11h
kube-apiserver-ip-10-0-172-171.us-east-2.compute.internal      2/2     Running     0          11h

Retrieved the cert being served with:

openssl s_client -showcerts -servername api.xxia-0606.qe.devcluster.openshift.com -connect api.xxia-0606.qe.devcluster.openshift.com:6443 </dev/null | tee -a showcert.out 

and compared it to tls.crt in the secret api-certs in the namespace openshift-config.   The certificates did not match.   Verified in the apiserver CR that api-certs is still the servingCertificate.

I will include the openssl output and the secret output in the must-gather zip I will be linking shortly.

Comment 10 Mike Fiedler 2019-06-06 16:49:39 UTC
Moving this back ON_QA.   Using the correct hostname for the SNI version of the openssl showcerts does show that we are serving the correct cert:

openssl s_client -showcerts -servername api.xxia-test.qe.devcluster.openshift.com -connect api.xxia-0606.qe.devcluster.openshift.com:6443

Certificate chain
 0 s:C = US, ST = test, L = Default City, O = Default Company Ltd, CN = api.xxia-test.qe.devcluster.openshift.com
   i:C = US, ST = test, L = Default City, O = Default Company Ltd, CN = api.xxia-test.qe.devcluster.openshift.com

Comment 11 Mike Fiedler 2019-06-06 16:59:05 UTC
@deads verified the subject and signer from the cert and I was able to verify the expiration:

echo | openssl s_client -showcerts -servername api.xxia-test.qe.devcluster.openshift.com -connect api.xxia-0606.qe.devcluster.openshift.com:6443 2>/dev/null | openssl x509 -text
        Version: 1 (0x0)
        Serial Number:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = test, L = Default City, O = Default Company Ltd, CN = api.xxia-test.qe.devcluster.openshift.com
            Not Before: Jun  6 04:39:18 2019 GMT
            Not After : Jun  7 04:39:18 2019 GMT
        Subject: C = US, ST = test, L = Default City, O = Default Company Ltd, CN = api.xxia-test.qe.devcluster.openshift.com

Marking this VERIFIED on 4.1.0-0.nightly/release/4.1.0-0.nightly-2019-06-05-223716.   New certificate is served from disk without new rollout.

Comment 13 errata-xmlrpc 2019-06-19 06:45:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.