Bug 1622338 - Fresh overcloud deployment fails when tls is enabled.
Summary: Fresh overcloud deployment fails when tls is enabled.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: z3
: 13.0 (Queens)
Assignee: Juan Antonio Osorio
QA Contact: Pavan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-26 13:01 UTC by Alfredo Pizarro
Modified: 2018-11-13 22:29 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.7-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-13 22:28:47 UTC
Target Upstream Version:
Embargoed:
lmarsh: needinfo-
lmarsh: needinfo-


Attachments (Terms of Use)
Overcloud template files (16.97 KB, application/x-gzip)
2018-08-26 13:01 UTC, Alfredo Pizarro
no flags Details
overcloud event list output (389.81 KB, text/plain)
2018-08-29 23:19 UTC, Alfredo Pizarro
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 594003 0 'None' MERGED Redis: Mount certificates on TLS proxy 2020-07-07 03:05:44 UTC
Red Hat Product Errata RHBA-2018:3587 0 None None None 2018-11-13 22:29:30 UTC

Description Alfredo Pizarro 2018-08-26 13:01:53 UTC
Created attachment 1478771 [details]
Overcloud template files

Description of problem:

I'm deploying a basic OSP13 overcloud with 1 controller and 1 node. When I deploy the overcloud without tls enabled it works fine, but when I enable tls the deployment fails:

overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cda24974-cd2d-4cc6-8792-81679a4ca4c8
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "stdout: 28116606bb48e20bb94517d9a276a35933b0d4db243f0253b8b00c210d6fa410", 
            "stdout: 993a89884fb133691d67b7d0de9f243cb367c754d8f2171bec6e7a027d24df06", 
            "stdout: 420e723717f4d34b25d5315248cff41ba6ef15470971486b5954d7c9bab57162"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/a72e0e73-6968-40ee-beac-bd92ccfba549_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=6    changed=2    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |


In the controller node, I can check that the digital certificate for the public IP was correctly installed and CA certificates as well:

[root@control1 ~]# curl https://overcloud.testlab.local
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://overcloud.testlab.local/dashboard">here</a>.</p>
</body></html>

In the  controller and try to find related errors and it is always something like this:

Aug 26 03:24:46 control1 os-collect-config[2997]: quests/sessions.py\\\", line 518, in request\", \n        \"    resp = self.send(prep, **send_kwargs)\", \n        \"  File \\\"/usr/lib/python2.7/site-packages/requests/sessions.py\\\", line 639, in send\", \n        \"    r = adapter.send(request, **kwargs)\", \n        \"  File \\\"/usr/lib/python2.7/site-packages/requests/adapters.py\\\", line 488, in send\", \n        \"    raise ConnectionError(err, request=request)\", \n        \"ConnectionError: ('Connection aborted.', BadStatusLine(\\\"''\\\",))\",

This error  makes me think that a  python script is trying to connect using http instead https.

I have tried this implementation several times, recreating digital certificates, changing settings about ip or dns public certificates and the result and the error is always the same and in the same stage.

Here is my overcloud script:

#!/bin/bash
#-e /home/stack/templates/environments/inject-trust-anchor.yaml \

openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \
-r /home/stack/templates/roles_data.yaml \
-e /home/stack/templates/environments/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/neutron-ovn-dvr-ha.yaml \
-e /home/stack/templates/environments/node-info.yaml \
-e /home/stack/templates/environments/overcloud_images.yaml \
-e /home/stack/templates/enable-tls.yaml \
-e /home/stack/templates/environments/inject-trust-anchor.yaml \
-e /home/stack/templates/inject-trust-anchor-hiera.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml \
-e /home/stack/templates/cloudname.yaml \
-e /home/stack/templates/environments/hostname-map.yaml \
-e /home/stack/templates/dashboard-ip.yaml

I'm attaching my overcloud template files to this ticket. 

Version-Release number of selected component (if applicable):
OSP 13: 
openstack-tripleo-heat-templates-8.0.2-43.el7ost.noarch


How reproducible:


Steps to Reproduce:
1.Install a basic OSP 13 overcloud (1 node / 1 controller) with enable-tls option enabled. 
2. Try the same overcloud settings without enable-tls options and it should work fine.
3.

Actual results:
It fails with this same error:
overcloud.AllNodesDeploySteps.ControllerDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cda24974-cd2d-4cc6-8792-81679a4ca4c8
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "stdout: 28116606bb48e20bb94517d9a276a35933b0d4db243f0253b8b00c210d6fa410", 
            "stdout: 993a89884fb133691d67b7d0de9f243cb367c754d8f2171bec6e7a027d24df06", 
            "stdout: 420e723717f4d34b25d5315248cff41ba6ef15470971486b5954d7c9bab57162"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/a72e0e73-6968-40ee-beac-bd92ccfba549_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=6    changed=2    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |


Expected results:
It should work 

Additional info:

Comment 1 Juan Antonio Osorio 2018-08-29 16:33:35 UTC
Can you pass the output of

openstack stack failures list --long overcloud

?

Comment 2 Alfredo Pizarro 2018-08-29 23:19:50 UTC
Created attachment 1479605 [details]
overcloud event list output

Attaching the required output.

Comment 3 Juan Antonio Osorio 2018-09-06 14:12:45 UTC
Alfredo, the gnocchi update issue is fixed by this commit https://review.openstack.org/#/c/594003/ is it included already? It merged two weeks ago.

Comment 4 Alfredo Pizarro 2018-09-06 14:51:07 UTC
Juan Antonio, I'm afraid it is not included in this setup. I will update the containers images and I will try again. I will let you know ASAP.

Comment 5 Alfredo Pizarro 2018-09-07 14:00:35 UTC
Hello Juan Antonio,  I updated all the containers images (including gnocchi) and the error persisted, the deployment failed with the same error, but I noticed that I was missing an update for openstack-tripleo-heat-templates in director.  I updated heat templates and now it's working:
Original version: openstack-tripleo-heat-templates-8.0.2-43.el7ost.noarch
Update: openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch

I deployed again with the updated version of openstack-tripleo-heat-templates using the same setup and it finished successfully.

Comment 16 errata-xmlrpc 2018-11-13 22:28:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587


Note You need to log in before you can comment on or make changes to this bug.