Bug 1427003

Summary: Failed to add masters if openshift_master_ca_certificate is defined
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: InstallerAssignee: Andrew Butcher <abutcher>
Status: CLOSED ERRATA QA Contact: Gan Huang <ghuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.5.0CC: abutcher, aos-bugs, ghuang, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously the specified openshift_master_ca_certificate file was not deployed when performing a master scaleup. The scaleup playbooks have been updated to ensure that this certificate is deployed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-12 19:02:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gan Huang 2017-02-27 03:08:13 UTC
Description of problem:
Installer failed to add masters if openshift_master_ca_certificate was defined in inventory hosts file previously.

Version-Release number of selected component (if applicable):
openshift-ansible-3.5.14-1

How reproducible:
always

Steps to Reproduce:
1. Trigger HA deplyment by specifying CA and named certificates.

# cat inventory_hosts

[OSEv3:vars]

openshift_master_ca_certificate={'certfile': '/root/1487924908-02-Feb-24-Feb-2017/rootCA.pem', 'keyfile': '/root/1487924908-02-Feb-24-Feb-2017/rootCA.key'}
openshift_master_named_certificates=[{"certfile": "/root/1487924908-02-Feb-24-Feb-2017/openshift-146.lab.sjc.redhat.com.crt", "keyfile": "/root/1487924908-02-Feb-24-Feb-2017/openshift-146.lab.sjc.redhat.com.key", "cafile": "/root/1487924908-02-Feb-24-Feb-2017/rootCA.pem"}]

2. Add a master after the installation above by running byo/openshift-master/scaleup.yml


Actual results:
TASK [openshift_ca : Deploy master ca certificate] *****************************
failed: [openshift-103.lab.sjc.redhat.com -> openshift-145.lab.sjc.redhat.com] (item={u'dest': u'ca.crt', u'src': u'/root/1487924908-02-Feb-24-Feb-2017/rootCA.pem'}) => {
    "checksum": "4aceb0b3eafd23b67b604412f521ae3796e9a141", 
    "failed": true, 
    "gid": 0, 
    "group": "root", 
    "item": {
        "dest": "ca.crt", 
        "src": "/root/1487924908-02-Feb-24-Feb-2017/rootCA.pem"
    }, 
    "mode": "0644", 
    "owner": "root", 
    "path": "/etc/origin/master/ca.crt", 
    "secontext": "system_u:object_r:etc_t:s0", 
    "size": 1578, 
    "src": "rootCA.pem", 
    "state": "hard", 
    "uid": 0
}

MSG:

src file does not exist, use "force=yes" if you really want to create the link: /etc/origin/master/rootCA.pem
failed: [openshift-103.lab.sjc.redhat.com -> openshift-145.lab.sjc.redhat.com] (item={u'dest': u'ca.key', u'src': u'/root/1487924908-02-Feb-24-Feb-2017/rootCA.key'}) => {
    "checksum": "de30aa334bb181f697b688dea113480c5118f65c", 
    "failed": true, 
    "gid": 0, 
    "group": "root", 
    "item": {
        "dest": "ca.key", 
        "src": "/root/1487924908-02-Feb-24-Feb-2017/rootCA.key"
    }, 
    "mode": "0644", 
    "owner": "root", 
    "path": "/etc/origin/master/ca.key", 
    "secontext": "system_u:object_r:etc_t:s0", 
    "size": 1679, 
    "src": "rootCA.key", 
    "state": "hard", 
    "uid": 0
}

MSG:

src file does not exist, use "force=yes" if you really want to create the link: /etc/origin/master/rootCA.key

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-master/scaleup.retry

PLAY RECAP *********************************************************************


Expected results:
No errors

Additional info:
Installer succeed to add masters after commenting openshift_master_ca_certificate

Comment 2 Gan Huang 2017-02-27 09:29:12 UTC
Hmm, probably it's not a issue about scale up playbook. I also often hit such issue when re-run the BYO playbook. This can be fixed by uninstall first.

Comment 3 Andrew Butcher 2017-02-27 20:23:22 UTC
Hey Gan,

Is the path "/root/1487924908-02-Feb-24-Feb-2017/rootCA.pem" local to the host where ansible is being ran or does that file exist on the remote systems?

Comment 4 Gan Huang 2017-02-28 07:49:10 UTC
Yes, "/root/1487924908-02-Feb-24-Feb-2017/rootCA.pem" was only on the ansible host.

Have experienced the same issue when re-run BYO playbook. I'm going to attach the logs and inventory hosts file.

Comment 7 openshift-github-bot 2017-03-02 02:47:27 UTC
Commit pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/4a3e61e035e42a260e0bf59d1e0c891dc004d50d
Merge pull request #3528 from abutcher/master-scaleup

Bug 1427003: Failed to add masters if openshift_master_ca_certificate is defined

Comment 9 Gan Huang 2017-03-02 12:46:51 UTC
Reproduced with openshift-ansible-3.5.18-1.git.0.01f8d4a.el7.noarch

No such issue in openshift-ansible-3.5.20-1.git.0.5a5fcd5.el7.noarch

Move to verified

Comment 11 errata-xmlrpc 2017-04-12 19:02:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903