Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1258497

Summary:	Heat fails to deploy compute and controller machines due to a random installation failure
Product:	Red Hat OpenStack	Reporter:	Raoul Scarazzini <rscarazz>
Component:	openstack-heat	Assignee:	Zane Bitter <zbitter>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Amit Ugol <augol>
Severity:	medium	Docs Contact:
Priority:	low
Version:	unspecified	CC:	fdinitto, mburns, michele, rhel-osp-director-maint, sbaker, shardy, srevivo
Target Milestone:	---	Keywords:	TestOnly, ZStream
Target Release:	7.0 (Kilo)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-heat-2015.1.1-1.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-06-23 18:19:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1255759
Bug Blocks:

Description Raoul Scarazzini 2015-08-31 13:54:44 UTC

Description of problem:


Version-Release number of selected component (if applicable): openstack-heat-engine-2015.1.0-6.el7ost.noarch


How reproducible: provisioning the machine via beaker and following the procedure described here http://10.33.11.10/pub/director-training/. In this case we are deploying 3 controllers and 2 compute VMs on the same HW (access details below)


Steps to Reproduce:
1. Follow the steps described in the training until "Lab 6: Deployment of Overcloud"
2. Follow the commands and finally launch the overcloud deploy:

openstack overcloud deploy --templates \
    --ntp-server 10.5.26.10 --control-scale 3 --compute-scale 2 \
    --neutron-tunnel-types vxlan --neutron-network-type vxlan
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates

Note that we are using openstack-heat-*-2015.1.0-6.el7ost.noarch so we should not be incurring in https://bugzilla.redhat.com/show_bug.cgi?id=1249628


Actual results:
After 240 minutes on most tries it ends with this error:
ERROR: openstack ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required

like described on https://bugzilla.redhat.com/show_bug.cgi?id=1238133

Expected results:
The procedure must end in success and then the instances of the overcloud are accessible.

Additional info:

This is the state of the heat stack list:

[stack@undercloud ~]$ heat stack-list --show-nested -f "status=FAILED"                                                                                                                                                                       │
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+       │| id                                   | stack_name                                                                                                    | stack_status  | creation_time        | parent                               |       │
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+       │
| 10d07858-33ae-4f5f-b4cb-5bcae3dc1e34 | overcloud                                                                                                     | CREATE_FAILED | 2015-08-31T07:41:06Z | None                                 |       │
| d07e8081-16f6-4c6c-81ed-43fa2e921036 | overcloud-ComputeNodesPostDeployment-czulbc2sdahk                                                             | CREATE_FAILED | 2015-08-31T07:51:08Z | 10d07858-33ae-4f5f-b4cb-5bcae3dc1e34 |       │
| 6e75e355-5251-4a1e-a428-a5c89672590a | overcloud-ComputeNodesPostDeployment-czulbc2sdahk-ComputePuppetDeployment-qrgibzdktgm7                        | CREATE_FAILED | 2015-08-31T07:51:11Z | d07e8081-16f6-4c6c-81ed-43fa2e921036 |       │
| 9570d403-86b7-431d-8d18-c8003439565d | overcloud-ControllerNodesPostDeployment-gw77e2sdbfzl                                                          | CREATE_FAILED | 2015-08-31T07:51:38Z | 10d07858-33ae-4f5f-b4cb-5bcae3dc1e34 |       │
| 7b0ffb95-d173-4fc3-b47f-91993de71061 | overcloud-ControllerNodesPostDeployment-gw77e2sdbfzl-ControllerOvercloudServicesDeployment_Step4-fpyaj7n43qxu | CREATE_FAILED | 2015-08-31T07:59:45Z | 9570d403-86b7-431d-8d18-c8003439565d |       │
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+       │

[stack@undercloud ~]$ heat resource-list --nested-depth 15 overcloud | grep FAILED
| ComputeNodesPostDeployment                  | d07e8081-16f6-4c6c-81ed-43fa2e921036          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED   | 2015-08-31T07:41:07Z |                                             |
| ControllerNodesPostDeployment               | 9570d403-86b7-431d-8d18-c8003439565d          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2015-08-31T07:41:07Z |                                             |
| ComputePuppetDeployment                     | 6e75e355-5251-4a1e-a428-a5c89672590a          | OS::Heat::StructuredDeployments                   | CREATE_FAILED   | 2015-08-31T07:51:08Z | ComputeNodesPostDeployment                  |
| 0                                           | 6ac65046-87b3-44c7-95e8-7556443fe219          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-31T07:51:11Z | ComputePuppetDeployment                     |
| 1                                           | dac1e789-1152-4200-934b-48d52a24a38c          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-31T07:51:11Z | ComputePuppetDeployment                     |
| ControllerOvercloudServicesDeployment_Step4 | 7b0ffb95-d173-4fc3-b47f-91993de71061          | OS::Heat::StructuredDeployments                   | CREATE_FAILED   | 2015-08-31T07:51:39Z | ControllerNodesPostDeployment               |
| 2                                           | d1bca05e-1e10-486b-b3f7-355c077fe983          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-31T07:59:45Z | ControllerOvercloudServicesDeployment_Step4 |

Some additional heat resource-show output can be found here: 
http://file.rdu.redhat.com/~mbaldess/bz-heat.txt

The failed machine is available at: mrg-09.mpc.lab.eng.bos.redhat.com
pwd: redhat

The undercloud is available via "ssh stack@undercloud"

Comment 3 Zane Bitter 2015-09-02 21:28:07 UTC

There was a second Heat race condition bug, 1255759. Possibly this could be caused by that.

Comment 4 Zane Bitter 2015-09-09 21:58:29 UTC

Please retest with the race condition fix from bug 1255759.

Comment 6 Lon Hohberger 2016-03-15 19:42:46 UTC

According to our records, this should be resolved by openstack-heat-2015.1.2-10.el7ost

Comment 7 Amit Ugol 2016-03-22 03:52:41 UTC

I'll keep this one open for a little while. If no one hits this issue I'll close it.

Comment 8 Amit Ugol 2016-05-18 08:46:42 UTC

The issues as originally described no longer seem to effect installation.
If there will be new issues, they probably new ones so open a new tracker bug.