Bug 1661781

Summary: [OSP 13] overcloud deployment breaks at "overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution"
Product: Red Hat OpenStack Reporter: Alexander Stafeyev <astafeye>
Component: openstack-tripleoAssignee: Brent Eagles <beagles>
Status: CLOSED CURRENTRELEASE QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: astafeye, bbonguar, bcafarel, cgoncalves, ihrachys, jagee, lpeer, majopela, mburns
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-10 13:41:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1613576, 1622182    

Description Alexander Stafeyev 2018-12-23 09:43:51 UTC
Openstack with octavia and tls everywhere deployment is failing . 

(undercloud) [stack@undercloud-0 ~]$ openstack server list 
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| c85a9aff-4084-4e3b-bde1-89c08da7dd08 | compute-0    | ACTIVE | ctlplane=192.168.24.8  | overcloud-full | compute    |
| bb8b624a-1c75-4089-b2e7-c075222548b5 | controller-2 | ACTIVE | ctlplane=192.168.24.7  | overcloud-full | controller |
| df53d788-a315-4a56-b786-b1d818042730 | controller-1 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
| 5fb4bb27-122d-4bb7-9ed9-2a5ce1fedbfa | compute-1    | ACTIVE | ctlplane=192.168.24.16 | overcloud-full | compute    |
| 9aded407-0817-410d-acbc-3b7c37883426 | compute-2    | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | compute    |
| 8a005a8e-1bda-4450-89e6-693e896a02f0 | controller-0 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
(undercloud) [stack@undercloud-0 ~]$ 

(undercloud) [stack@undercloud-0 ~]$ openstack stack list 
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| 4159472f-7e55-44d8-8574-583d44301174 | overcloud  | b42122f63343497a9bddcabed87fbc71 | CREATE_FAILED | 2018-12-23T08:20:03Z | None         |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 
13   -p 2018-12-13.4
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ rpm -qa | grep openstack-tripleo-heat-templates 
openstack-tripleo-heat-templates-8.0.7-21.el7ost.noarch
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ openstack stack resource list  overcloud | grep FA
| AllNodesDeploySteps                       | ed78c5e5-da32-4cb8-8a2e-025bf27630b2                      | OS::TripleO::PostDeploySteps                     | CREATE_FAILED   | 2018-12-23T08:20:04Z |
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh 
#!/bin/bash

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/nodes_data.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /home/stack/virt/extra_templates.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /home/stack/virt/cloud-names.yaml \
--log-file overcloud_deployment_4.log
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: 1942c495-92b3-42cb-837f-3187c2c3f20f
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step5_Execution: ERROR

Comment 2 Bernard Cafarelli 2019-01-02 10:31:35 UTC
From mistral executor.log:
Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.
Unable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host=\'overcloud.redhat.local\', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(\'<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f591b7ec890>: Failed to establish a new connection: [Errno -2] Name or service not known\',))

So this looks like the same issue as bug #1659078 for 13

Comment 10 Bruna Bonguardo 2019-04-03 14:46:43 UTC
The patch was merged. 
I just triggered the CI job to check if it passes.

If the CI job passes - I will check if a Polarion test was automatically triggered.

Comment 12 Bruna Bonguardo 2019-04-10 13:41:14 UTC
The CI job passed and a Polarion test was automatically triggered successfully.

Closing the bug.