Bug 1585978

Summary: Redeploy openshift ca playbook failed
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: InstallerAssignee: Russell Teague <rteague>
Status: CLOSED CURRENTRELEASE QA Contact: Gaoyun Pei <gpei>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: antonio, aos-bugs, jokerman, judd, mmccomas, wsun
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Corrected redeploy-openshift-ca.yml playbook to reference the correct node client certificate file, node/client-ca.crt.
Story Points: ---
Clone Of:
: 1628546 (view as bug list) Environment:
Last Closed: 2018-12-21 15:16:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1628546    
Attachments:
Description Flags
redeploy-openshift-ca.yml -v none

Description Gaoyun Pei 2018-06-05 08:42:00 UTC
Description of problem:
Run redeploy-openshift-ca.yml playbook against an ocp-3.10 cluster, playbook failed as below:

TASK [Update node client kubeconfig CA data] ********************************************************************************************************************************
fatal: [ec2-34-207-246-134.compute-1.amazonaws.com]: FAILED! => {"changed": false, "failed": true, "msg": "[Errno 2] No such file or directory: '/etc/origin/node/system:node:ip-172-18-0-157.ec2.internal.kubeconfig'"}
fatal: [ec2-52-90-247-129.compute-1.amazonaws.com]: FAILED! => {"changed": false, "failed": true, "msg": "[Errno 2] No such file or directory: '/etc/origin/node/system:node:ip-172-18-11-10.ec2.internal.kubeconfig'"}


Version-Release number of the following components:
openshift-ansible-3.10.0-0.58.0.git.0.d8f6377.el7.noarch


How reproducible:
Always

Steps to Reproduce:
ansible-playbook -i host/310 -v /usr/share/ansible/openshift-ansible/playbooks/openshift-master/redeploy-openshift-ca.yml


Actual results:
Ansible inventory file and full log with "-v" could be found in attachment.


Expected results:

Additional info:

Comment 4 Antonio Guillen 2018-06-09 23:58:56 UTC
Created attachment 1449529 [details]
redeploy-openshift-ca.yml -v

[agd@folio13 openshift-ansible]$ git describe
openshift-ansible-3.10.0-0.63.0-60-g7b4160215
[agd@folio13 openshift-ansible]$ ansible-playbook --version | head -n1
ansible-playbook 2.5.3

Comment 6 Judd Maltin 2018-09-07 14:05:59 UTC
Same bug bit me.   Blocks updating my learning.redhat.com online class.

rpm -qa | grep openshift

openshift-ansible-playbooks-3.10.21-1.git.0.6446011.el7.noarch
atomic-openshift-clients-3.10.14-1.git.0.ba8ae6d.el7.x86_64
openshift-ansible-roles-3.10.21-1.git.0.6446011.el7.noarch
atomic-openshift-3.10.14-1.git.0.ba8ae6d.el7.x86_64
openshift-ansible-3.10.21-1.git.0.6446011.el7.noarch
openshift-ansible-docs-3.10.21-1.git.0.6446011.el7.noarch

Comment 7 Judd Maltin 2018-09-07 15:24:04 UTC
Fix.  https://github.com/openshift/openshift-ansible/pull/9958

Comment 8 Wei Sun 2018-09-10 08:48:25 UTC
The PR 9958 has been merged to penshift-ansible-3.11.0-0.33.0

Comment 9 Gaoyun Pei 2018-09-11 05:10:56 UTC
Waiting for openshift-ansible 3.11.0-0.33.0 build

Comment 10 Gaoyun Pei 2018-09-12 10:24:47 UTC
Tested with openshift-ansible-3.11.0-0.33.0.git.0.9052676. The CA redeployment playbook could run without such error.

But I see installer is still doing some outdated steps in https://github.com/newgoliath/openshift-ansible/blob/master/playbooks/openshift-master/private/redeploy-openshift-ca.yml#L230

The CA file used in servingInfo.clientCA is client-ca.crt per node-config.yaml, no {{ openshift.common.config_base }}/node/ca.crt file generated in an ocp-3.10+ fresh installation. 

So the result is client-ca.crt didn't get updated, but a useless ca.crt was copied to each node. Pls correct me if anything missed. Thanks.

Comment 11 Russell Teague 2018-09-12 18:46:52 UTC
master: https://github.com/openshift/openshift-ansible/pull/10034

Comment 12 Scott Dodson 2018-09-13 01:46:47 UTC
https://github.com/openshift/openshift-ansible/pull/10041 release-3.11

Comment 13 Russell Teague 2018-09-13 12:04:49 UTC
openshift-ansible-3.11.2-1

Comment 14 Gaoyun Pei 2018-09-14 03:50:13 UTC
Verify this bug with openshift-ansible-3.11.5-1.git.0.5a01a3c.el7_5.noarch

After run openshift CA redeployment playbook
On master:
/etc/origin/master/ca.crt updated to a new CA cert file, the new CA added into ca-bundle.crt and client-ca-bundle.crt, CA data of admin.kubeconfig and openshift-master.kubeconfig got updated with new ca-bundle.crt content. 

On nodes:
ca-bundle.crt copied to each node as the new /etc/origin/node/client-ca.crt, CA data of node.kubeconfig got updated with new ca-bundle.crt content.

Master and node service are all running well and app's creation test passed.

Comment 15 Luke Meyer 2018-12-21 15:16:35 UTC
Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content.