Bug 1597002

Description of problem

The fast forward upgrade command fails with ssh authentication errors:

(undercloud) [stack@undercloud ~]$ openstack overcloud ffwd-upgrade run --yes

The command that seems to be running in the backend is this one:

Running cmd (subprocess): /usr/bin/tripleo-ansible-inventory --stack overcloud --ansible_ssh_user heat-admin --static-yaml-inventory /home/stack/tripleo-ansible-inventory.yaml
CMD "/usr/bin/tripleo-ansible-inventory --stack overcloud --ansible_ssh_user heat-admin --static-yaml-inventory /home/stack/tripleo-ansible-inventory.yaml" returned: 0 in 4.019s


The stack currently is in UPDATE_COMPLETE state:

(undercloud) [stack@undercloud ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| id                                   | stack_name | stack_status    | creation_time        | updated_time         | project                          |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| f5d898c4-8fd1-41cf-bb4b-b63372b46669 | overcloud  | UPDATE_COMPLETE | 2018-06-30T01:07:16Z | 2018-07-01T04:30:31Z | 5d1395eabc8d48ae9ad6681576bf6904 |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+

The command for preparing the nodes for fast forward upgrade completed successfully without any errors:

(undercloud) [stack@undercloud ~]$ openstack overcloud ffwd-upgrade prepare --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ~/templates/network-environment.yaml  -e ~/templates/storage-environment.yaml -e ~/templates/overcloud_images.yaml -e ~/templates/node-info.yaml -e ~/templates/registration.yaml --yes


Here's a snippet of the error that is noticed:


u'fatal: [192.168.24.10]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.10\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.16]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.16\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.7]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.7\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.12]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.12\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.18]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.18\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.9]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.9\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.8]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.8\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',
 u'fatal: [192.168.24.14]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Could not create directory \'/home/mistral/.ssh\'.\\r\\nWarning: Permanently added \'192.168.24.14\' (ECDSA) to the list of known hosts.\\r\\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\\r\\n", "unreachable": true}',

The ansible_ssh_user seems to have been set correctly in the playbook:

(undercloud) [stack@undercloud ~]$ grep -i ansible_ssh_user tripleo-ansible-inventory.yaml
    ansible_ssh_user: heat-admin
    ansible_ssh_user: heat-admin
    ansible_ssh_user: heat-admin
....


I notice that there was another bug for this where the ansible_ssh_user was set to an incorrect value https://bugzilla.redhat.com/show_bug.cgi?id=1576079

This was fixed in: python-tripleoclient-9.2.1-11

Version-Release number of selected component (if applicable):

(undercloud) [stack@undercloud ~]$ rpm -qa | grep tripleo
ansible-tripleo-ipsec-8.1.1-0.20180308133440.8f5369a.el7ost.noarch
openstack-tripleo-heat-templates-compat-7.0.9-8.1.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-38.el7ost.noarch
openstack-tripleo-ui-8.3.1-3.el7ost.noarch
openstack-tripleo-0.0.8-0.3.4de13b3git.el7ost.noarch
python-tripleoclient-9.2.1-12.el7ost.noarch
openstack-tripleo-common-devtools-8.6.1-20.el7ost.noarch
openstack-tripleo-common-containers-8.6.1-20.el7ost.noarch
openstack-tripleo-common-8.6.1-20.el7ost.noarch
puppet-tripleo-8.3.2-8.el7ost.noarch
openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.0-2.el7ost.noarch
openstack-tripleo-validations-8.4.1-5.el7ost.noarch
openstack-tripleo-common-container-base-8.6.1-20.el7ost.noarch


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

sosreport from the undercloud node, output of the command, ansible inventory that was generated, custom templates used.

Please let me know if further information is required.