Bug 1572874

Description of problem:
Setup an ocp-3.9 cluster, set the server time to 2 years later on each host to make signed ocp certs expired. Run redeploy-certificates.yml to update the expired certs.


After playbook finished, atomic-openshift-master-controllers service failed to be started:

[root@qe-gpei-392-master-etcd-nfs-1 service-catalog]# journalctl  -f -u atomic-openshift-master-controllers.service
-- Logs begin at Fri 2020-05-01 00:00:00 EDT. --
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: I0501 00:28:22.346358   34392 controller_utils.go:1026] Caches are synced for scheduler controller
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: I0501 00:28:22.346379   34392 leaderelection.go:175] attempting to acquire leader lease  kube-system/kube-scheduler...
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: I0501 00:28:22.349528   34392 leaderelection.go:243] lock is held by qe-gpei-392-master-etcd-nfs-1_9e3f4f89-8b61-11ea-80f0-42010af0000f and has not yet expired
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: I0501 00:28:22.349542   34392 leaderelection.go:180] failed to acquire lease kube-system/kube-scheduler
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: I0501 00:28:22.764841   34392 request.go:1076] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'x509: certificate has expired or is not yet valid'
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: Trying to reach: 'https://172.30.186.202:443/apis/servicecatalog.k8s.io/v1beta1''
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 atomic-openshift-master-controllers[34392]: F0501 00:28:22.792148   34392 start_master.go:655] Error starting "openshift.io/cluster-quota-reconciliation" (failed to discover resources: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: an error on the server ("Error: 'x509: certificate has expired or is not yet valid'\nTrying to reach: 'https://172.30.186.202:443/apis/servicecatalog.k8s.io/v1beta1'") has prevented the request from succeeding)
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 systemd[1]: atomic-openshift-master-controllers.service: main process exited, code=exited, status=255/n/a
May 01 00:28:22 qe-gpei-392-master-etcd-nfs-1 systemd[1]: Unit atomic-openshift-master-controllers.service entered failed state.



Since service catalog is installed by default in 3.9, so seems we have to also update the server-catalog certs as well when redeploying the certs.


Version-Release number of the following components:
openshift-ansible-3.9.27-1.git.0.52e35b5.el7.noarch


How reproducible:
100%

Steps to Reproduce:
1.Run playbooks/redeploy-certificates.yml against an ocp-3.9 cluster with all signed ocp certs expired.
ansible-playbook -i host/39 /usr/share/ansible/openshift-ansible/playbooks/redeploy-certificates.yml


Actual results:
atomic-openshift-master-controllers service not running


Expected results:
The certs files for service-catalog should also be updated.


Additional info: