Bug 1702401

Summary:	Service catalog does not have a redeploy-certificate playbook
Product:	OpenShift Container Platform	Reporter:	Joseph Callen <jcallen>
Component:	Installer	Assignee:	Joseph Callen <jcallen>
Installer sub component:	openshift-ansible	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	gpei, jiazha
Version:	3.11.0
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	Feature: Playbook to recreate the certificate for the service catalog Reason: The certificates for the service catalog need to rotated like another other component of OpenShift Result: The certificates for the service catalog will be update.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-23 19:56:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joseph Callen 2019-04-23 16:53:37 UTC

Description of problem:
The service catalog does not have a `redeploy-certificates.yml` playbook.  If the ca were to be replaced this may cause issues with the service catalog.



Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Joseph Callen 2019-04-25 12:49:04 UTC

https://github.com/openshift/openshift-ansible/pull/11554

Comment 2 Russell Teague 2019-06-11 19:32:49 UTC

New PR: https://github.com/openshift/openshift-ansible/pull/11681

Comment 10 Jian Zhang 2019-06-20 02:49:10 UTC

Hi, Joseph

Yes, you're right. It works well in a fresh cluster. Maybe someone changed something in that old cluster. Thanks! Details:
mac:openshift-ansible jianzhang$ ansible-playbook -i qe-inventory-host-file playbooks/redeploy-certificates.yml -vvv
...
PLAY RECAP **************************************************************************************************************************************************
localhost                  : ok=16   changed=0    unreachable=0    failed=0   
vm-10-0-77-86.hosted.upshift.rdu2.redhat.com : ok=231  changed=78   unreachable=0    failed=0   


INSTALLER STATUS ********************************************************************************************************************************************
Initialization  : Complete (0:01:05)
Thursday 20 June 2019  10:13:04 +0800 (0:00:35.639)       0:12:29.888 ********* 
=============================================================================== 
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------- 37.53s
/Users/jianzhang/project/openshift-ansible/playbooks/init/basic_facts.yml:2 --------------------------------------------------------------------------------
openshift_service_catalog : Verify that the apiserver is running ------------------------------------------------------------------------------------ 37.04s
/Users/jianzhang/project/openshift-ansible/roles/openshift_service_catalog/tasks/restart_pods.yml:19 -------------------------------------------------------
openshift_service_catalog : Verify that the controller-manager is running --------------------------------------------------------------------------- 35.98s
/Users/jianzhang/project/openshift-ansible/roles/openshift_service_catalog/tasks/restart_pods.yml:40 -------------------------------------------------------
template_service_broker : Verify that the apiserver is running -------------------------------------------------------------------------------------- 35.64s
/Users/jianzhang/project/openshift-ansible/roles/template_service_broker/tasks/restart_pods.yml:9 ----------------------------------------------------------
Remove generated certificates ----------------------------------------------------------------------------------------------------------------------- 29.03s
/Users/jianzhang/project/openshift-ansible/playbooks/openshift-master/private/certificates-backup.yml:28 ---------------------------------------------------
ansible_service_broker : Verify that the ASB is running --------------------------------------------------------------------------------------------- 24.32s
/Users/jianzhang/project/openshift-ansible/roles/ansible_service_broker/tasks/restart_pods.yml:20 ----------------------------------------------------------
openshift_control_plane : verify API server --------------------------------------------------------------------------------------------------------- 18.66s
/Users/jianzhang/project/openshift-ansible/roles/openshift_control_plane/handlers/main.yml:13 --------------------------------------------------------------
openshift_control_plane : verify API server --------------------------------------------------------------------------------------------------------- 17.14s
/Users/jianzhang/project/openshift-ansible/roles/openshift_control_plane/handlers/main.yml:13 --------------------------------------------------------------
openshift_console : Copy console templates to temp directory ---------------------------------------------------------------------------------------- 15.64s
/Users/jianzhang/project/openshift-ansible/roles/openshift_console/tasks/install.yml:19 --------------------------------------------------------------------
openshift_console : Waiting for console rollout to complete ----------------------------------------------------------------------------------------- 14.76s
/Users/jianzhang/project/openshift-ansible/roles/openshift_console/tasks/start.yml:2 -----------------------------------------------------------------------
template_service_broker : Remove apiserver pods ----------------------------------------------------------------------------------------------------- 14.32s
/Users/jianzhang/project/openshift-ansible/roles/template_service_broker/tasks/restart_pods.yml:2 ----------------------------------------------------------
etcd : restart etcd --------------------------------------------------------------------------------------------------------------------------------- 12.01s
/Users/jianzhang/project/openshift-ansible/roles/etcd/tasks/restart.yml:2 ----------------------------------------------------------------------------------
openshift_service_catalog : Generating API Server keys ---------------------------------------------------------------------------------------------- 11.61s
/Users/jianzhang/project/openshift-ansible/roles/openshift_service_catalog/tasks/generate_certs.yml:29 -----------------------------------------------------
Wait for master API to come back online ------------------------------------------------------------------------------------------------------------- 11.51s
/Users/jianzhang/project/openshift-ansible/playbooks/openshift-node/private/restart.yml:54 -----------------------------------------------------------------
Remove web console pods ----------------------------------------------------------------------------------------------------------------------------- 10.87s
/Users/jianzhang/project/openshift-ansible/playbooks/openshift-web-console/private/redeploy-certificates.yml:16 --------------------------------------------
etcd : Retrieve etcd ca cert tarball ---------------------------------------------------------------------------------------------------------------- 10.34s
/Users/jianzhang/project/openshift-ansible/roles/etcd/tasks/certificates/fetch_server_certificates_from_ca.yml:165 -----------------------------------------
etcd : template -------------------------------------------------------------------------------------------------------------------------------------- 9.45s
/Users/jianzhang/project/openshift-ansible/roles/etcd/tasks/certificates/deploy_ca.yml:32 ------------------------------------------------------------------
etcd : Unarchive cert tarball ------------------------------------------------------------------------------------------------------------------------ 9.27s
/Users/jianzhang/project/openshift-ansible/roles/etcd/tasks/certificates/fetch_server_certificates_from_ca.yml:149 -----------------------------------------
etcd : copy ------------------------------------------------------------------------------------------------------------------------------------------ 8.95s
/Users/jianzhang/project/openshift-ansible/roles/etcd/tasks/certificates/deploy_ca.yml:63 ------------------------------------------------------------------
openshift_hosted : Create OpenShift router ----------------------------------------------------------------------------------------------------------- 8.46s
/Users/jianzhang/project/openshift-ansible/roles/openshift_hosted/tasks/router.yml:85 ----------------------------------------------------------------------

These secrets(asb-client, asb-tls, apiserver-serving-cert, templateservicebroker-client) have been updated as expected. As below:
[root@qe-phunt-preserve-merrn-1 ~]# oc get secret -n openshift-ansible-service-broker
NAME                         TYPE                                  DATA      AGE
asb-client                   kubernetes.io/service-account-token   4         5m
asb-client-dockercfg-rrplx   kubernetes.io/dockercfg               1         6h
asb-client-token-8t55n       kubernetes.io/service-account-token   4         6h
asb-client-token-hdp2p       kubernetes.io/service-account-token   4         6h
asb-dockercfg-lzknv          kubernetes.io/dockercfg               1         6h
asb-registry-auth            Opaque                                2         6h
asb-tls                      kubernetes.io/tls                     2         5m
asb-token-gg4g7              kubernetes.io/service-account-token   4         6h
...

[root@qe-phunt-preserve-merrn-1 ~]# oc get secret -n openshift-template-service-broker
NAME                                           TYPE                                  DATA      AGE
apiserver-dockercfg-qrprr                      kubernetes.io/dockercfg               1         6h
apiserver-serving-cert                         kubernetes.io/tls                     2         5m
apiserver-token-67drq                          kubernetes.io/service-account-token   4         6h
apiserver-token-dpdmr                          kubernetes.io/service-account-token   4         6h
builder-dockercfg-5288k                        kubernetes.io/dockercfg               1         6h
builder-token-jxlbd                            kubernetes.io/service-account-token   4         6h
builder-token-mfw2z                            kubernetes.io/service-account-token   4         6h
default-dockercfg-gfzpm                        kubernetes.io/dockercfg               1         6h
default-token-7pz2t                            kubernetes.io/service-account-token   4         6h
default-token-bvjv8                            kubernetes.io/service-account-token   4         6h
deployer-dockercfg-7v548                       kubernetes.io/dockercfg               1         6h
deployer-token-bmghk                           kubernetes.io/service-account-token   4         6h
deployer-token-dcdfk                           kubernetes.io/service-account-token   4         6h
templateservicebroker-client                   kubernetes.io/service-account-token   4         5m
...

And, the ASB/TSB service works well!
[root@qe-phunt-preserve-merrn-1 ~]# curl -vvv --cacert /etc/origin/master/service-signer.crt https://asb.openshift-ansible-service-broker.svc:1338
* About to connect() to asb.openshift-ansible-service-broker.svc port 1338 (#0)
*   Trying 172.30.7.188...
* Connected to asb.openshift-ansible-service-broker.svc (172.30.7.188) port 1338 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/origin/master/service-signer.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=asb.openshift-ansible-service-broker.svc
* 	start date: Jun 20 02:11:19 2019 GMT
* 	expire date: Jun 19 02:11:20 2021 GMT
* 	common name: asb.openshift-ansible-service-broker.svc
* 	issuer: CN=openshift-service-serving-signer@1560996300
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: asb.openshift-ansible-service-broker.svc:1338
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 20 Jun 2019 02:17:57 GMT
< Content-Length: 162
< 
{
  "paths": [
    "/apis",
    "/healthz",
    "/healthz/ping",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/metrics",
    "/osb/"
  ]
* Connection #0 to host asb.openshift-ansible-service-broker.svc left intact


[root@qe-phunt-preserve-merrn-1 ~]# curl -vvv --cacert /etc/origin/master/service-signer.crt https://apiserver.openshift-template-service-broker.svc:443
* About to connect() to apiserver.openshift-template-service-broker.svc port 443 (#0)
*   Trying 172.30.241.218...
* Connected to apiserver.openshift-template-service-broker.svc (172.30.241.218) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/origin/master/service-signer.crt
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=apiserver.openshift-template-service-broker.svc
* 	start date: Jun 20 02:12:05 2019 GMT
* 	expire date: Jun 19 02:12:06 2021 GMT
* 	common name: apiserver.openshift-template-service-broker.svc
* 	issuer: CN=openshift-service-serving-signer@1560996300
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: apiserver.openshift-template-service-broker.svc
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 20 Jun 2019 02:31:57 GMT
< Content-Length: 214
< 
{
  "paths": [
    "/apis",
    "/brokers/template.openshift.io/v2",
    "/healthz",
    "/healthz/log",
    "/healthz/ping",
    "/healthz/poststarthook/template-service-broker-synctemplates",
    "/metrics"
  ]
* Connection #0 to host apiserver.openshift-template-service-broker.svc left intact

Verify it, thanks!

mac:openshift-ansible jianzhang$ ansible --version
ansible 2.6.17.post0
  config file = /Users/jianzhang/project/openshift-ansible/ansible.cfg
  configured module search path = [u'/Users/jianzhang/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible-2.6.17.post0-py2.7.egg/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15 (default, Nov 27 2018, 21:40:55) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]

Comment 12 errata-xmlrpc 2019-07-23 19:56:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1753