While using an OSP13 puddle and doing a deployment which uses ceph-ansible to deploy an overcloud to use an external ceph cluster as described in documentation [1], the deployment fails on WorkflowTasks_Step2_Execution OS::Mistral::ExternalResource before ceph-ansible even runs. During the ceph-install workflow the enable_ssh_admin task [2] called the tripleo.access.v1.enable_ssh_admin workbook which failed on the task create_admin_via_nova [3]. Error may be seen by running the following on the undercloud: mistral task-get-result cd02e39d-6505-4e01-b06b-5f7218ae8369| jq . | sed -e 's/\\n/\n/g' -e 's/\\"/"/g' The output of the above is at: http://paste.openstack.org/show/693495/ Note that this bug as similar symptoms to 1552327 as originally reported but the issue happens before ceph-ansible runs and with OSP13 using ceph-ansible-3.1.0-0.1.beta3.el7.noarch, which contains the ceph-ansible 0.29 patch was basically a backport from 3.1.b4. [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/integrating_an_overcloud_with_an_existing_red_hat_ceph_cluster [2] https://github.com/openstack/tripleo-common/blob/master/workbooks/ceph-ansible.yaml#L56-L60 [3] https://github.com/openstack/tripleo-common/blob/master/workbooks/access.yaml#L89
/var/log/messages on controller node (192.168.213.201) shows that the tripleo-admin user was created BUT that heat was unable to communicate this back to the undercloud because of SSL hand shake error: Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [INFO] {"deploy_stdout": "\nPLAY [localhost] ***************************************************************\n\nTASK [Gather ing Facts] *********************************************************\nok: [localhost]\n\nTASK [create user tripleo-admin] ***********************************************\nchanged: [localhost]\n\nTASK [grant admi n rights to user tripleo-admin] ********************************\nchanged: [localhost]\n\nTASK [ensure .ssh dir exists for user tripleo-admin] ***************************\nchanged: [localhost]\n\nTASK [ensure au thorized_keys file exists for user tripleo-admin] ***************\nchanged: [localhost]\n\nTASK [authorize TripleO Mistral key for user tripleo-admin] ********************\nchanged: [localhost]\n\nPLAY RECAP *** ******************************************************************\nlocalhost : ok=6 changed=5 unreachable=0 failed=0 \n\n", "deploy_stderr": "", "deploy_status_code": 0} Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [DEBUG] [2018-04-09 16:15:07,047] (heat-config) [DEBUG] Running ansible-playbook -i localhost, /var/lib/heat-config/heat-con fig-ansible/8dc0ec12-e105-404e-af40-5bdde7543cc8_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/8dc0ec12-e105-404e-af40-5bdde7543cc8_variables.json Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,962] (heat-config) [INFO] Return code 0 Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,962] (heat-config) [INFO] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: PLAY [localhost] *************************************************************** Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [Gathering Facts] ********************************************************* Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: ok: [localhost] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [create user tripleo-admin] *********************************************** Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [grant admin rights to user tripleo-admin] ******************************** Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [ensure .ssh dir exists for user tripleo-admin] *************************** Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [ensure authorized_keys file exists for user tripleo-admin] *************** Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [authorize TripleO Mistral key for user tripleo-admin] ******************** Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost] Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: PLAY RECAP ********************************************************************* Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: localhost : ok=6 changed=5 unreachable=0 failed=0 Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,962] (heat-config) [INFO] Completed /var/lib/heat-config/heat-config-ansible/8dc0ec12-e105-404e-af40-5bdde7543cc8_playbook.yaml Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible Apr 9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/8dc0ec12-e105-404e-af40-5bdde7543cc8.json < /var/lib/heat-config/deployed/8dc0ec12-e105-404e-af40-5bdde7543cc8.notify.json Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:12,572] (heat-config) [INFO] Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:12,573] (heat-config) [ERROR] Error running heat-config-notify. [1] Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:12,573] (heat-config) [ERROR] [2018-04-09 16:15:12,517] (heat-config-notify) [DEBUG] Signaling to https://192.168.213.2:13808/v1/AUTH_3bc5279c95b74b46bf0362a0929ad107/create_admin-4d0f4c6f-5843-440e-9322-09fb0999fb76/dcc32759-9e21-43e7-8125-75d2907d1628?temp_url_sig=e7194f36c40f1627f2d614af3ce47145fd9b563b&temp_url_expires=1523322887 via PUT Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: Traceback (most recent call last): Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/bin/heat-config-notify", line 179, in <module> Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: sys.exit(main(sys.argv, sys.stdin)) Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/bin/heat-config-notify", line 126, in main Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: headers={'content-type': 'application/json'}) Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in put Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: return self.request('PUT', url, data=data, **kwargs) Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: resp = self.send(prep, **send_kwargs) Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: r = adapter.send(request, **kwargs) Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 512, in send Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: raise SSLError(e, request=request) Apr 9 16:15:12 rhosp-ctrl0 os-collect-config: requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
ROOT CAUSE The task which created the tripleo-admin user on the overcloud was unable to signal to the undercloud that it succeeded because it got an SSL handshake error. The undercloud didn't receive a message back from the overcloud so it timed out.
I've seen this error in the past and addressed it by either: 1. Adding "generate_service_certificate = false" to my undercloud.conf 2. Using inject-trust-anchor.yaml as per https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html It looks like neither of the above are the case in this deployment [1] I'm closing this as not a bug as things are working as expected given the inputs/outputs. [1] A. undercloud.conf contained: #undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem generate_service_certificate = true certificate_generation_ca = local B. deployment command was: openstack overcloud deploy --templates --ntp-server 10.5.27.10 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --environment-directory /home/stack/mytemplates/env --roles-file /home/stack/mytemplates/rhosp-roles-data.yaml --networks-file /home/stack/mytemplates/rhosp-network-data.yaml --stack rhosp --debug --log-file overcloudDeploy.log No trust anchor settings in /home/stack/mytemplates/env (undercloud) [stack@rhosp-director env]$ ll total 40 -rw-r--r--. 1 stack stack 1935 Apr 9 19:28 25-rhosp-networks.yaml -rw-rw-r--. 1 stack stack 4602 Apr 9 19:28 30-ips-from-pools.yaml -rw-r--r--. 1 stack stack 764 Apr 9 19:28 35-rhosp-IP-pools.yaml -rw-r--r--. 1 stack stack 1349 Apr 9 19:28 50-rhosp-ext-ceph.yaml -rw-r--r--. 1 stack stack 879 Apr 9 19:28 60-sat6-registration.yaml -rw-r--r--. 1 stack stack 351 Apr 9 19:28 70-rhosp-hostmap.yaml -rw-rw-r--. 1 stack stack 6641 Apr 9 19:28 90-rhosp-rhos13-images.yaml -rw-r--r--. 1 stack stack 556 Apr 9 19:28 99-rhosp-misc.yaml (undercloud) [stack@rhosp-director env]$
*** Bug 1637013 has been marked as a duplicate of this bug. ***