Deployment with ceph fails with: "WorkflowTasks_Step2_Execution: ERROR" "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'. Host key verification failed." Environment: openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch instack-undercloud-7.4.2-0.20171010064304.el7ost.noarch puppet-ceph-2.4.2-0.20170927195215.718a5ff.el7ost.noarch ceph-ansible-3.0.3-1.el7cp.noarch openstack-tripleo-heat-templates-7.0.3-0.20171014102841.el7ost.noarch Steps to reproduce: Attempt to deploy with: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/inject-trust-anchor-hiera.yaml \ -e /home/stack/rhos12.yaml Result: The deployment fails: CREATE_IN_PROGRESS state changed\n2017-10-20 00:52:12Z [overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution]: CREATE_FAILED resources.WorkflowTasks_Step2_Execution: ERROR\n2017-10-20 00:52:13Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR\n2017-10-20 00:52:13Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR\n2017-10-20 00:52:13Z [overcloud]: CREATE_FAILED Resource CREATE failed: resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR\n\n Stack overcloud CREATE_FAILED \n\novercloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:\n resource_type: OS::Mistral::ExternalResource\n physical_resource_id: 99c12b7f-ed4e-494b-9da3-f66c59ccce53\n status: CREATE_FAILED\n status_reason: |\n resources.WorkflowTasks_Step2_Execution: ERROR" } Digging through mistral logs I see: onnect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}]} 2017-10-19 17:56:37,061 p=23479 u=mistral | failed: [192.168.24.12] (item=192.168.24.22) => {"item": "192.168.24.22", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true} 2017-10-19 17:56:37,113 p=23479 u=mistral | failed: [192.168.24.12] (item=192.168.24.6) => {"item": "192.168.24.6", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true} 2017-10-19 17:56:37,113 p=23479 u=mistral | fatal: [192.168.24.12]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_item_result": true, "item": "192.168.24.11", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.12", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.9", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.21", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.16", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.18", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.22", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}, {"_ansible_item_result": true, "item": "192.168.24.6", "msg": "Failed to connect to the host via ssh: Could not create directory '/home/mistral/.ssh'.\r\nHost key verification failed.\r\n", "unreachable": true}]}
Created attachment 1341126 [details] mistral logs from UC.
We launch ansible-playbook with: --ssh-common-args "-o StrictHostKeyChecking=no" --ssh-extra-args "-o UserKnownHostsFile=/dev/null" but those options seem to be ignored as ssh tries to create the known_host file in the home dir instead. This wasn't the case before. We're debugging the issue with Marius, it looks like we're hitting a bug in ansible 2.4.0 [1] which makes it ignore the above options. I tested manually the proposed fix [2] and it worked. Previously we were using ansible 2.3.2 and the upstream bug seems to have been fixed in 2.4.1, which is not released yet. I think as a temporary measure the only option is to: # yum downgrade ansible to reinstal 2.3.2. 1. https://github.com/ansible/ansible/issues/30720 2. https://github.com/ansible/ansible/pull/31326
Verified. Environment: openstack-tripleo-common-7.6.3-0.20171010234828.el7ost.noarch The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462