Bug 1575445

Summary: FFU: openstack overcloud ffwd-upgrade run doesn't work with custom non-default --ssh-user option
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: rhosp-directorAssignee: Marios Andreou <mandreou>
Status: CLOSED DUPLICATE QA Contact: Amit Ugol <augol>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: dbecker, mandreou, mburns, morazi
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-09 11:01:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2018-05-07 00:36:07 UTC
Description of problem:
FFU: openstack overcloud ffwd-upgrade run doesn't work with custom non-default --ssh-user:

openstack overcloud ffwd-upgrade run --ssh-user admin --yes

Warning! The TripleO Fast Forward Upgrade workflow is currently considered under development. In particular invocations of the ffwd-upgrade cli should be initially limited to development/test environments. Once and if you decide to use ffwd-upgrade in production, ensure you are adequately prepared with valid backup of your current deployment state.
Continuing fast forward upgrade
Started Mistral Workflow tripleo.package_update.v1.update_nodes. Execution ID: 13c40cc7-41d3-428e-ac79-96f4d64580f5
Waiting for messages on queue 'ffwdupgrade' with no timeout.
[u'Using /tmp/ansible-mistral-actionhN4lEx/ansible.cfg as config file',
 u' [WARNING]: Skipping unexpected key (hostvars) in group (_meta), only "vars",',
 u'"children" and "hosts" are valid',
 u'',
 u'PLAY [overcloud] ***************************************************************',
 u'',
 u'TASK [Gathering Facts] *********************************************************',
 u'fatal: [192.168.0.14]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.14\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.20]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.20\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.16]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.16\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.28]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.28\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.21]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.21\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.18]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.18\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.13]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.13\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.0.23]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.0.23\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'',
 u'PLAY RECAP *********************************************************************',
 u'192.168.0.13               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.14               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.16               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.18               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.20               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.21               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.23               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'192.168.0.28               : ok=0    changed=0    unreachable=1    failed=0   ',
 u'']


Version-Release number of selected component (if applicable):
python-tripleoclient-9.2.1-3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 with custom ssh user used for overcloud nodes:

openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \
--overcloud-ssh-user admin \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/centralci-prod-customisations.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/scheduler_hints_env.yaml \
-e ~/openstack_deployment/environments/ips-from-pool-all.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \

2. Upgrade to OSP13 via fast forward procedure

3. Run the ffwd-upgrade run step:
openstack overcloud ffwd-upgrade run --ssh-user admin --yes

Actual results:

Fails because it cannot reach the overcloud nodes which means the user passed via --ssh-user(admin in this case) was not used.

Expected results:
Command succeeds fine and it uses the custom user name passed by the operator via --ssh-user. 

Additional info:
We can actually see in the generated inventory.yaml that the default 'heat-admin' was used not the custom one:

[root@undercloud-0 stack]# grep ansible_ssh_user /tmp/ansible-mistral-action815Z96/inventory.yaml 
    ansible_ssh_user: heat-admin
    ansible_ssh_user: heat-admin
    ansible_ssh_user: heat-admin

Comment 2 Marios Andreou 2018-05-08 11:10:04 UTC
thanks mcornea i think i see the bug going to post a review shortly

Comment 4 Marios Andreou 2018-05-09 11:01:51 UTC
closing this as duplicate of BZ 1576079 - ccamacho landed https://review.openstack.org/#/c/566960/ which is duplicate for https://review.openstack.org/#/c/566859/ (tracked here) so lets use the other BZ please re-open if you disagree thanks

*** This bug has been marked as a duplicate of bug 1576079 ***