Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1565861

Summary:	enable_ssh_admin task unable to signal undercloud due to SSL handshake error
Product:	Red Hat OpenStack	Reporter:	John Fulton <johfulto>
Component:	ceph-ansible	Assignee:	John Fulton <johfulto>
Status:	CLOSED NOTABUG	QA Contact:	Yogev Rabl <yrabl>
Severity:	low	Docs Contact:
Priority:	low
Version:	13.0 (Queens)	CC:	akaris, ebarrera, gfidente, gkadam, ukalifon
Target Milestone:	ga	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-11 03:01:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Fulton 2018-04-11 00:12:37 UTC

While using an OSP13 puddle and doing a deployment which uses ceph-ansible to deploy an overcloud to use an external ceph cluster as described in documentation [1], the deployment fails on WorkflowTasks_Step2_Execution OS::Mistral::ExternalResource before ceph-ansible even runs. 

During the ceph-install workflow the enable_ssh_admin task [2] called the tripleo.access.v1.enable_ssh_admin workbook which failed on the task create_admin_via_nova [3]. 

Error may be seen by running the following on the undercloud: 

mistral task-get-result cd02e39d-6505-4e01-b06b-5f7218ae8369| jq . | sed -e 's/\\n/\n/g' -e 's/\\"/"/g'

The output of the above is at:  http://paste.openstack.org/show/693495/

Note that this bug as similar symptoms to 1552327 as originally reported but the issue happens before ceph-ansible runs and with OSP13 using ceph-ansible-3.1.0-0.1.beta3.el7.noarch, which contains the ceph-ansible 0.29 patch was basically a backport from 3.1.b4. 


[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/integrating_an_overcloud_with_an_existing_red_hat_ceph_cluster

[2] https://github.com/openstack/tripleo-common/blob/master/workbooks/ceph-ansible.yaml#L56-L60

[3] https://github.com/openstack/tripleo-common/blob/master/workbooks/access.yaml#L89

Comment 3 John Fulton 2018-04-11 02:17:01 UTC

/var/log/messages on controller node (192.168.213.201) shows that the tripleo-admin user was created BUT that heat was unable to communicate this back to the undercloud because of SSL hand shake error:

Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [INFO] {"deploy_stdout": "\nPLAY [localhost] ***************************************************************\n\nTASK [Gather
ing Facts] *********************************************************\nok: [localhost]\n\nTASK [create user tripleo-admin] ***********************************************\nchanged: [localhost]\n\nTASK [grant admi
n rights to user tripleo-admin] ********************************\nchanged: [localhost]\n\nTASK [ensure .ssh dir exists for user tripleo-admin] ***************************\nchanged: [localhost]\n\nTASK [ensure au
thorized_keys file exists for user tripleo-admin] ***************\nchanged: [localhost]\n\nTASK [authorize TripleO Mistral key for user tripleo-admin] ********************\nchanged: [localhost]\n\nPLAY RECAP ***
******************************************************************\nlocalhost                  : ok=6    changed=5    unreachable=0    failed=0   \n\n", "deploy_stderr": "", "deploy_status_code": 0}
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [DEBUG] [2018-04-09 16:15:07,047] (heat-config) [DEBUG] Running ansible-playbook -i localhost, /var/lib/heat-config/heat-con
fig-ansible/8dc0ec12-e105-404e-af40-5bdde7543cc8_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/8dc0ec12-e105-404e-af40-5bdde7543cc8_variables.json
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,962] (heat-config) [INFO] Return code 0
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,962] (heat-config) [INFO]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: PLAY [localhost] ***************************************************************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [Gathering Facts] *********************************************************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: ok: [localhost]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [create user tripleo-admin] ***********************************************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [grant admin rights to user tripleo-admin] ********************************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [ensure .ssh dir exists for user tripleo-admin] ***************************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [ensure authorized_keys file exists for user tripleo-admin] ***************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: TASK [authorize TripleO Mistral key for user tripleo-admin] ********************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: changed: [localhost]
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: PLAY RECAP *********************************************************************
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: localhost                  : ok=6    changed=5    unreachable=0    failed=0
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,962] (heat-config) [INFO] Completed /var/lib/heat-config/heat-config-ansible/8dc0ec12-e105-404e-af40-5bdde7543cc8_playbook.yaml
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible
Apr  9 16:15:11 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:11,966] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/8dc0ec12-e105-404e-af40-5bdde7543cc8.json < /var/lib/heat-config/deployed/8dc0ec12-e105-404e-af40-5bdde7543cc8.notify.json
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:12,572] (heat-config) [INFO]
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:12,573] (heat-config) [ERROR] Error running heat-config-notify. [1]
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: [2018-04-09 16:15:12,573] (heat-config) [ERROR] [2018-04-09 16:15:12,517] (heat-config-notify) [DEBUG] Signaling to https://192.168.213.2:13808/v1/AUTH_3bc5279c95b74b46bf0362a0929ad107/create_admin-4d0f4c6f-5843-440e-9322-09fb0999fb76/dcc32759-9e21-43e7-8125-75d2907d1628?temp_url_sig=e7194f36c40f1627f2d614af3ce47145fd9b563b&temp_url_expires=1523322887 via PUT
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: Traceback (most recent call last):
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/bin/heat-config-notify", line 179, in <module>
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: sys.exit(main(sys.argv, sys.stdin))
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/bin/heat-config-notify", line 126, in main
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: headers={'content-type': 'application/json'})
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in put
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: return self.request('PUT', url, data=data, **kwargs)
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: resp = self.send(prep, **send_kwargs)
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: r = adapter.send(request, **kwargs)
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 512, in send
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: raise SSLError(e, request=request)
Apr  9 16:15:12 rhosp-ctrl0 os-collect-config: requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

Comment 5 John Fulton 2018-04-11 02:30:12 UTC

ROOT CAUSE

The task which created the tripleo-admin user on the overcloud was unable to signal to the undercloud that it succeeded because it got an SSL handshake error.

The undercloud didn't receive a message back from the overcloud so it timed out.

Comment 6 John Fulton 2018-04-11 03:01:04 UTC

I've seen this error in the past and addressed it by either:

1. Adding "generate_service_certificate = false" to my undercloud.conf
2. Using inject-trust-anchor.yaml as per https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html

It looks like neither of the above are the case in this deployment [1]

I'm closing this as not a bug as things are working as expected given the inputs/outputs. 

[1] 

A. undercloud.conf contained: 

#undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem 
generate_service_certificate = true
certificate_generation_ca = local

B. deployment command was:

openstack overcloud deploy --templates --ntp-server 10.5.27.10 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --environment-directory /home/stack/mytemplates/env --roles-file /home/stack/mytemplates/rhosp-roles-data.yaml --networks-file /home/stack/mytemplates/rhosp-network-data.yaml --stack rhosp --debug --log-file overcloudDeploy.log

No trust anchor settings in /home/stack/mytemplates/env

(undercloud) [stack@rhosp-director env]$ ll
total 40
-rw-r--r--. 1 stack stack 1935 Apr  9 19:28 25-rhosp-networks.yaml
-rw-rw-r--. 1 stack stack 4602 Apr  9 19:28 30-ips-from-pools.yaml
-rw-r--r--. 1 stack stack  764 Apr  9 19:28 35-rhosp-IP-pools.yaml
-rw-r--r--. 1 stack stack 1349 Apr  9 19:28 50-rhosp-ext-ceph.yaml
-rw-r--r--. 1 stack stack  879 Apr  9 19:28 60-sat6-registration.yaml
-rw-r--r--. 1 stack stack  351 Apr  9 19:28 70-rhosp-hostmap.yaml
-rw-rw-r--. 1 stack stack 6641 Apr  9 19:28 90-rhosp-rhos13-images.yaml
-rw-r--r--. 1 stack stack  556 Apr  9 19:28 99-rhosp-misc.yaml
(undercloud) [stack@rhosp-director env]$

Comment 7 John Fulton 2018-10-08 16:07:39 UTC

*** Bug 1637013 has been marked as a duplicate of this bug. ***