Bug 1460970

Summary: [3.4] Redeploy CA will try to restart services when certs are expired, causing failure.
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: InstallerAssignee: Andrew Butcher <abutcher>
Status: CLOSED ERRATA QA Contact: Gaoyun Pei <gpei>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.4.1CC: abutcher, aos-bugs, jokerman, mmccomas, rhowe, smunilla
Target Milestone: ---   
Target Release: 3.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1452367 Environment:
Last Closed: 2017-06-29 13:33:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1452367, 1463773    
Bug Blocks:    

Comment 3 Gaoyun Pei 2017-06-15 09:19:49 UTC
Test with openshift-ansible-3.4.99-1.git.0.84718ab.el7.noarch, redeploy CA playbook failed as:

PLAY [Validate configuration for rolling restart] ******************************

TASK [setup] *******************************************************************
fatal: [ec2-54-209-69-217.compute-1.amazonaws.com]: FAILED! => {
    "failed": true
}

MSG:

The conditional check '('expired' not in hostvars | oo_select_keys(groups['oo_masters_to_config']) | oo_collect('check_results.check_results.ocp_certs') | oo_collect('health', {'path':hostvars[groups.oo_first_master.0].openshift.common.config_base ~ "/master/master.server.crt"})) and ('expired' not in hostvars | oo_select_keys(groups['oo_masters_to_config']) | oo_collect('check_results.check_results.ocp_certs') | oo_collect('health', {'path':hostvars[groups.oo_first_master.0].openshift.common.config_base ~ "/master/ca-bundle.crt"}))' failed. The error was: 'list' object has no attribute 'get'
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-openshift-ca.retry

Comment 7 Gaoyun Pei 2017-06-28 05:15:49 UTC
Verify this bug with openshift-ansible-3.4.109-1.git.0.576c8dd.el7.noarch

When openshift certs expired, redeploy openshift CA cert
ansible-playbook -i host /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-openshift-ca.yml
Redeploy openshift CA playbook will update openshift CA cert and skip restart master/node service since expired cert detected. 

Redeploy etcd CA cert
ansible-playbook -i host /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-etcd-ca.yml
Redeploy openshift CA playbook will update etcd CA cert and skip restart etcd/master service since expired cert detected. 

Redeploy openshift certs next:
ansible-playbook -i host /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-certificates.yml
This playbook will generate new certs and restart etcd/master/docker/node service.

Then all the certs were replaced by new certs, ocp env works well again.

Comment 9 errata-xmlrpc 2017-06-29 13:33:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1666