Bug 1477718

Summary: Install mixed CRS environment failed due to glusterfs_heketi_ssh_keyfile didn't copy to first master host
Product: OpenShift Container Platform Reporter: Wenkai Shi <weshi>
Component: InstallerAssignee: Jose A. Rivera <jarrpa>
Status: CLOSED ERRATA QA Contact: Wenkai Shi <weshi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6.0CC: aos-bugs, jarrpa, jokerman, mmccomas, xtian
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Copy SSH private key to master node.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:06:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Wenkai Shi 2017-08-02 17:19:11 UTC
Description of problem:
Reference to "inventory/byo/hosts.byo.glusterfs.mixed.example" 's example, install a mixed CRS environment. Before installation, prepare ssh private key for heketi to communicate to the GlusterFS nodes in "/root/glusterfs_key", this file is only on ansible host. During installation installer didn't copy it to first master, then installer failed in "Create heketi config secret" step because the keyfile didn't on first master host. 

Version-Release number of the following components:
openshift-ansible-3.6.172.0.2-1.git.0.5bd2286.el7

How reproducible:
100%

Steps to Reproduce:
1. Deploy mixed CRS environment, reference to "inventory/byo/hosts.byo.glusterfs.mixed.example" 's example.
# cat hosts
[OSEv3:children]
masters
nodes
glusterfs

[OSEv3:vars]
...
openshift_storage_glusterfs_is_native=false
openshift_storage_glusterfs_heketi_is_native=true
openshift_storage_glusterfs_heketi_executor=ssh
openshift_storage_glusterfs_heketi_ssh_keyfile=/root/glusterfs_key

[masters]
master-1.example.com

[nodes]
master-1.example.com
node-1.example.com

[glusterfs]
glusterfs-1.example.com glusterfs_devices="['/dev/vsda']"
glusterfs-2.example.com glusterfs_devices="['/dev/vsda']"
glusterfs-3.example.com glusterfs_devices="['/dev/vsda']"

2.
3.

Actual results:
# ansible-playbook -i hosts -v /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
...
TASK [openshift_storage_glusterfs : Create heketi config secret] ***************
Wednesday 02 August 2017  14:56:39 +0000 (0:00:00.059)       0:11:49.661 ****** 
fatal: [master-1.example.com]: FAILED! => {
    "changed": false, 
    "failed": true
}

MSG:

{u'returncode': 1, u'cmd': u'/usr/bin/oc secrets new heketi-storage-config-secret heketi.json=/tmp/openshift-glusterfs-ansible-q9TD6U/heketi.json private_key=/root/glusterfs_key -n glusterfs', u'results': {}, u'stderr': u'error: error reading /root/glusterfs_key: no such file or directory\n', u'stdout': u''}


Expected results:
Before create heketi config secret, the key file should be copy to first master host.

Additional info:

Comment 1 Jose A. Rivera 2017-08-02 18:29:45 UTC
Fixed in upstream PR: https://github.com/openshift/openshift-ansible/pull/4982

Comment 3 Wenkai Shi 2017-08-04 04:22:58 UTC
Verified with version openshift-ansible-3.6.173.0.5-1.git.0.74d5acc.el7, installer could create heketi config secret succeed.

# ansible-playbook -i hosts -v /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
...
TASK [openshift_storage_glusterfs : Copy initial heketi resource files] ********
Friday 04 August 2017  03:50:35 +0000 (0:00:00.676)       0:09:09.777 ********* 
changed: [hmaster-1.example.com] => (item=deploy-heketi-template.yml) => {
    "changed": true, 
    "checksum": "7ed00a463467a668783f6e34a429c046d0a0ffdc", 
    "dest": "/tmp/openshift-glusterfs-ansible-8esDkR/deploy-heketi-template.yml", 
    "gid": 0, 
    "group": "root", 
    "item": "deploy-heketi-template.yml", 
    "md5sum": "1f821c579382b5fc0b155ebf55cdedc9", 
    "mode": "0644", 
    "owner": "root", 
    "secontext": "unconfined_u:object_r:admin_home_t:s0", 
    "size": 4120, 
    "src": "/root/.ansible/tmp/ansible-tmp-1501818635.83-267549938333782/source", 
    "state": "file", 
    "uid": 0
}

TASK [openshift_storage_glusterfs : Create heketi topology secret] *************
Friday 04 August 2017  03:50:36 +0000 (0:00:00.379)       0:09:10.157 ********* 
changed: [master-1.example.com] => {
    "changed": true, 
    "results": {
        "cmd": "/usr/bin/oc secrets new heketi-storage-topology-secret topology.json=/tmp/openshift-glusterfs-ansible-8esDkR/topology.json -n glusterfs", 
        "results": {}, 
        "returncode": 0
    }, 
    "state": "present"
}
...

Comment 7 errata-xmlrpc 2017-11-28 22:06:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188