Bug 1258497 - Heat fails to deploy compute and controller machines due to a random installation failure
Heat fails to deploy compute and controller machines due to a random installa...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
unspecified
x86_64 Linux
low Severity medium
: ---
: 7.0 (Kilo)
Assigned To: Zane Bitter
Amit Ugol
: TestOnly, ZStream
Depends On: 1255759
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-31 09:54 EDT by Raoul Scarazzini
Modified: 2016-06-23 14:19 EDT (History)
7 users (show)

See Also:
Fixed In Version: openstack-heat-2015.1.1-1.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-23 14:19:11 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Raoul Scarazzini 2015-08-31 09:54:44 EDT
Description of problem:


Version-Release number of selected component (if applicable): openstack-heat-engine-2015.1.0-6.el7ost.noarch


How reproducible: provisioning the machine via beaker and following the procedure described here http://10.33.11.10/pub/director-training/. In this case we are deploying 3 controllers and 2 compute VMs on the same HW (access details below)


Steps to Reproduce:
1. Follow the steps described in the training until "Lab 6: Deployment of Overcloud"
2. Follow the commands and finally launch the overcloud deploy:

openstack overcloud deploy --templates \
    --ntp-server 10.5.26.10 --control-scale 3 --compute-scale 2 \
    --neutron-tunnel-types vxlan --neutron-network-type vxlan
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates

Note that we are using openstack-heat-*-2015.1.0-6.el7ost.noarch so we should not be incurring in https://bugzilla.redhat.com/show_bug.cgi?id=1249628


Actual results:
After 240 minutes on most tries it ends with this error:
ERROR: openstack ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required

like described on https://bugzilla.redhat.com/show_bug.cgi?id=1238133

Expected results:
The procedure must end in success and then the instances of the overcloud are accessible.

Additional info:

This is the state of the heat stack list:

[stack@undercloud ~]$ heat stack-list --show-nested -f "status=FAILED"                                                                                                                                                                       │
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+       │| id                                   | stack_name                                                                                                    | stack_status  | creation_time        | parent                               |       │
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+       │
| 10d07858-33ae-4f5f-b4cb-5bcae3dc1e34 | overcloud                                                                                                     | CREATE_FAILED | 2015-08-31T07:41:06Z | None                                 |       │
| d07e8081-16f6-4c6c-81ed-43fa2e921036 | overcloud-ComputeNodesPostDeployment-czulbc2sdahk                                                             | CREATE_FAILED | 2015-08-31T07:51:08Z | 10d07858-33ae-4f5f-b4cb-5bcae3dc1e34 |       │
| 6e75e355-5251-4a1e-a428-a5c89672590a | overcloud-ComputeNodesPostDeployment-czulbc2sdahk-ComputePuppetDeployment-qrgibzdktgm7                        | CREATE_FAILED | 2015-08-31T07:51:11Z | d07e8081-16f6-4c6c-81ed-43fa2e921036 |       │
| 9570d403-86b7-431d-8d18-c8003439565d | overcloud-ControllerNodesPostDeployment-gw77e2sdbfzl                                                          | CREATE_FAILED | 2015-08-31T07:51:38Z | 10d07858-33ae-4f5f-b4cb-5bcae3dc1e34 |       │
| 7b0ffb95-d173-4fc3-b47f-91993de71061 | overcloud-ControllerNodesPostDeployment-gw77e2sdbfzl-ControllerOvercloudServicesDeployment_Step4-fpyaj7n43qxu | CREATE_FAILED | 2015-08-31T07:59:45Z | 9570d403-86b7-431d-8d18-c8003439565d |       │
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+       │

[stack@undercloud ~]$ heat resource-list --nested-depth 15 overcloud | grep FAILED
| ComputeNodesPostDeployment                  | d07e8081-16f6-4c6c-81ed-43fa2e921036          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED   | 2015-08-31T07:41:07Z |                                             |
| ControllerNodesPostDeployment               | 9570d403-86b7-431d-8d18-c8003439565d          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2015-08-31T07:41:07Z |                                             |
| ComputePuppetDeployment                     | 6e75e355-5251-4a1e-a428-a5c89672590a          | OS::Heat::StructuredDeployments                   | CREATE_FAILED   | 2015-08-31T07:51:08Z | ComputeNodesPostDeployment                  |
| 0                                           | 6ac65046-87b3-44c7-95e8-7556443fe219          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-31T07:51:11Z | ComputePuppetDeployment                     |
| 1                                           | dac1e789-1152-4200-934b-48d52a24a38c          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-31T07:51:11Z | ComputePuppetDeployment                     |
| ControllerOvercloudServicesDeployment_Step4 | 7b0ffb95-d173-4fc3-b47f-91993de71061          | OS::Heat::StructuredDeployments                   | CREATE_FAILED   | 2015-08-31T07:51:39Z | ControllerNodesPostDeployment               |
| 2                                           | d1bca05e-1e10-486b-b3f7-355c077fe983          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-31T07:59:45Z | ControllerOvercloudServicesDeployment_Step4 |

Some additional heat resource-show output can be found here: 
http://file.rdu.redhat.com/~mbaldess/bz-heat.txt

The failed machine is available at: mrg-09.mpc.lab.eng.bos.redhat.com
pwd: redhat

The undercloud is available via "ssh stack@undercloud"
Comment 3 Zane Bitter 2015-09-02 17:28:07 EDT
There was a second Heat race condition bug, 1255759. Possibly this could be caused by that.
Comment 4 Zane Bitter 2015-09-09 17:58:29 EDT
Please retest with the race condition fix from bug 1255759.
Comment 6 Lon Hohberger 2016-03-15 15:42:46 EDT
According to our records, this should be resolved by openstack-heat-2015.1.2-10.el7ost
Comment 7 Amit Ugol 2016-03-21 23:52:41 EDT
I'll keep this one open for a little while. If no one hits this issue I'll close it.
Comment 8 Amit Ugol 2016-05-18 04:46:42 EDT
The issues as originally described no longer seem to effect installation.
If there will be new issues, they probably new ones so open a new tracker bug.

Note You need to log in before you can comment on or make changes to this bug.