Bug 1688610

PR addressing the problem is https://github.com/openshift/cluster-kube-apiserver-operator/pull/338

The PR landed.

Didn't hit the related bug 1688820 today with latest build. Will continue checking more next day.

Read above PR, get all 30 days certs. Collect these certs' namespace and secret in a file:
$ cat certs.txt
openshift-kube-apiserver-operator aggregator-client-signer
openshift-kube-apiserver aggregator-client
openshift-kube-apiserver localhost-serving-cert-certkey
openshift-kube-apiserver service-network-serving-certkey
openshift-kube-apiserver loadbalancer-serving-certkey
openshift-config-managed kube-controller-manager-client-cert-key
openshift-kube-controller-manager kube-controller-manager-client-cert-key
openshift-config-managed kube-scheduler-client-cert-key
openshift-kube-scheduler kube-scheduler-client-cert-key
openshift-kube-apiserver kube-apiserver-cert-syncer-client-cert-key

Then check their dates in today's build latest 4.0.0-0.nightly-2019-03-19-004004 by:
export IFS=$'\n'
for i in `cat certs.txt`
do
  NS=`echo $i | cut -d ' ' -f 1`
  SECRET=`echo $i | cut -d ' ' -f 2`
  rm -f tls.crt
  oc extract secret/$SECRET -n $NS --confirm > /dev/null
  echo "Check cert dates of $SECRET in project $NS:"
  echo "openssl x509 -noout --dates -in tls.crt"
  openssl x509 -noout --dates -in tls.crt
  echo
done

Got:
Check cert dates of aggregator-client-signer in project openshift-kube-apiserver-operator:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 04:44:46 2019 GMT
notAfter=Mar 20 04:44:46 2019 GMT

Check cert dates of aggregator-client in project openshift-kube-apiserver:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:03 2019 GMT
notAfter=Mar 20 04:44:46 2019 GMT

Check cert dates of localhost-serving-cert-certkey in project openshift-kube-apiserver:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:03 2019 GMT
notAfter=Apr 18 05:02:04 2019 GMT

Check cert dates of service-network-serving-certkey in project openshift-kube-apiserver:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:03 2019 GMT
notAfter=Apr 18 05:02:04 2019 GMT

Check cert dates of loadbalancer-serving-certkey in project openshift-kube-apiserver:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:06 2019 GMT
notAfter=Apr 18 05:02:07 2019 GMT

Check cert dates of kube-controller-manager-client-cert-key in project openshift-config-managed:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:04 2019 GMT
notAfter=Apr 18 05:02:05 2019 GMT

Check cert dates of kube-controller-manager-client-cert-key in project openshift-kube-controller-manager:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:04 2019 GMT
notAfter=Apr 18 05:02:05 2019 GMT

Check cert dates of kube-scheduler-client-cert-key in project openshift-config-managed:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:02 2019 GMT
notAfter=Apr 18 05:02:03 2019 GMT

Check cert dates of kube-scheduler-client-cert-key in project openshift-kube-scheduler:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:02 2019 GMT
notAfter=Apr 18 05:02:03 2019 GMT

Check cert dates of kube-apiserver-cert-syncer-client-cert-key in project openshift-kube-apiserver:
openssl x509 -noout --dates -in tls.crt
notBefore=Mar 19 05:02:08 2019 GMT
notAfter=Apr 18 05:02:09 2019 GMT


Most of these certs have 30 days validity except the two: aggregator-client-signer and aggregator-client which only have 1 day validity.
Maciej, does this mean these two still have bug?

This is not a bug, the initial cert validity is 1 day, only after that period we'll be getting the default 30 days certs. You need to keep the cluster running for a longer period to verify the certs.

I still hit the issue in https://bugzilla.redhat.com/show_bug.cgi?id=1688820 on  4.0.0-0.nightly-2019-03-19-004004 after leaving the cluster up overnight.  It has not been up 24h

Yeah, while testing the ability to shorten the cert rotation I've noticed that openshift apiserver died with errors from https://bugzilla.redhat.com/show_bug.cgi?id=1688820.
I'll continue debugging that tomorrow.

(In reply to Maciej Szulik from comment #5)
> This is not a bug, the initial cert validity is 1 day, only after that
> period we'll be getting the default 30 days certs. You need to keep the
> cluster running for a longer period to verify the certs.

Yes, now these two show 30 days too:
Check cert dates of aggregator-client-signer in project openshift-kube-apiserver-operator:
openssl x509 -noout --dates -in tls.crt                                                                                                
notBefore=Mar 20 14:57:38 2019 GMT                                                                                                     
notAfter=Apr 19 14:57:39 2019 GMT 
                                 
Check cert dates of aggregator-client in project openshift-kube-apiserver:
openssl x509 -noout --dates -in tls.crt                                                                                                
notBefore=Mar 20 14:57:37 2019 GMT                                                                                                     
notAfter=Apr 18 23:56:56 2019 GMT

I'm confused. The original comment said "Before beta 3, we need to increase the certification rotation interval to allow customers to shutdown instances. Looking at the default duration for 3.11 for nodes, it looks like we set --experimental-cluster-signing-duration for 1 year" ... but then everything I see in this bz and the attached PR seems to suggest that all we did was bump it to 30 days.

Are there any more plans to address the original issue? For those of us in the training space (creating courses and exams for the product), 30 days is simply not enough. In case it's not clear, we need to be able to shutdown and snapshot environments that will then be copied and spun up for many many many months to come.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758