Bug 1380701

Summary: osp-director-10: first attempt of openstack undercloud install fails over "timed out creating the default Deployment Plan" (second attempt to re-run succeed)
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: instack-undercloudAssignee: Dougal Matthews <dmatthew>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: dbecker, dmatthew, jcoufal, jjoyce, jschluet, jslagle, mburns, mcornea, morazi, rhel-osp-director-maint, sasha, slinaber, tvignaud
Target Milestone: rcKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: instack-undercloud-5.0.0-0.20160929132429.56e65af.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:06:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1337794    

Description Omri Hochman 2016-09-30 11:33:00 UTC 
osp-director-10: first attempt of openstack undercloud install fails over "timed out creating the default Deployment Plan" (second attempt to re-run succeed)  

Environment : 
-------------
instack-undercloud-5.0.0-0.20160923031412.3a70958.el7ost.noarch
instack-5.0.0-0.20160921202724.5aabf5c.el7ost.noarch
openstack-heat-engine-7.0.0-0.20160923054727.e4c4c56.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-api-7.0.0-0.20160923054727.e4c4c56.el7ost.noarch
puppet-heat-9.2.0-0.20160917075914.182371c.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160922100830.75c20d5.1.el7ost.noarch
openstack-heat-api-cfn-7.0.0-0.20160923054727.e4c4c56.el7ost.noarch
python-heat-agent-0.0.1-0.20160920204709.f123aa1.el7ost.noarch
python-heatclient-1.4.0-0.20160831084943.fb7802e.el7ost.noarch
openstack-heat-common-7.0.0-0.20160923054727.e4c4c56.el7ost.noarch
openstack-heat-templates-0.0.1-0.20160920204709.f123aa1.el7ost.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20160916023622.793a8d9.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch


Steps: 
(1) Attempt to deploy osp10 undercloud ( run openstack undercloud install)

Results :
----------
'openstack undercloud install'  fails at first attempt. 
( re-running the same command finish successfully)  


'openstack undercloud install' (fails) : 
----------------------------------------
09:34:22 ++ cut -f 2 -d ' '
09:34:24 ++ ls /usr/share/openstack-tripleo-common/workbooks/baremetal.yaml /usr/share/openstack-tripleo-common/workbooks/deployment.yaml /usr/share/openstack-tripleo-common/workbooks/plan_management.yaml /usr/share/openstack-tripleo-common/workbooks/validations.yaml
09:34:24 + for workbook in '$(ls /usr/share/openstack-tripleo-common/workbooks/*)'
09:34:24 + mistral workbook-create /usr/share/openstack-tripleo-common/workbooks/baremetal.yaml
09:34:30 +------------+----------------------------+
09:34:30 | Field      | Value                      |
09:34:30 +------------+----------------------------+
09:34:30 | Name       | tripleo.baremetal.v1       |
09:34:30 | Tags       | <none>                     |
09:34:30 | Created at | 2016-09-26 11:29:30.255381 |
09:34:30 | Updated at | None                       |
09:34:30 +------------+----------------------------+
09:34:30 + for workbook in '$(ls /usr/share/openstack-tripleo-common/workbooks/*)'
09:34:30 + mistral workbook-create /usr/share/openstack-tripleo-common/workbooks/deployment.yaml
09:34:32 +------------+----------------------------+
09:34:32 | Field      | Value                      |
09:34:32 +------------+----------------------------+
09:34:32 | Name       | tripleo.deployment.v1      |
09:34:32 | Tags       | <none>                     |
09:34:32 | Created at | 2016-09-26 11:29:33.058057 |
09:34:32 | Updated at | None                       |
09:34:32 +------------+----------------------------+
09:34:32 + for workbook in '$(ls /usr/share/openstack-tripleo-common/workbooks/*)'
09:34:32 + mistral workbook-create /usr/share/openstack-tripleo-common/workbooks/plan_management.yaml
09:34:37 +------------+----------------------------+
09:34:37 | Field      | Value                      |
09:34:37 +------------+----------------------------+
09:34:37 | Name       | tripleo.plan_management.v1 |
09:34:37 | Tags       | <none>                     |
09:34:37 | Created at | 2016-09-26 11:29:37.971731 |
09:34:37 | Updated at | None                       |
09:34:37 +------------+----------------------------+
09:34:37 + for workbook in '$(ls /usr/share/openstack-tripleo-common/workbooks/*)'
09:34:37 + mistral workbook-create /usr/share/openstack-tripleo-common/workbooks/validations.yaml
09:34:41 +------------+----------------------------+
09:34:41 | Field      | Value                      |
09:34:41 +------------+----------------------------+
09:34:41 | Name       | tripleo.validations.v1     |
09:34:41 | Tags       | <none>                     |
09:34:41 | Created at | 2016-09-26 11:29:42.341588 |
09:34:41 | Updated at | None                       |
09:34:41 +------------+----------------------------+
09:34:41 + sysctl -w net.ipv4.ip_forward=1
09:34:41 net.ipv4.ip_forward = 1
09:34:41 + echo net.ipv4.ip_forward=1
09:34:41 dib-run-parts Mon Sep 26 07:29:42 EDT 2016 98-undercloud-setup completed
09:34:41 dib-run-parts Mon Sep 26 07:29:42 EDT 2016 Running /usr/libexec/os-refresh-config/post-configure.d/99-refresh-completed
09:34:41 ++ os-apply-config --key completion-handle --type raw --key-default ''
09:34:41 [2016/09/26 07:29:42 AM] [WARNING] DEPRECATED: falling back to /var/run/os-collect-config/os_config_files.json
09:34:41 + HANDLE=
09:34:41 ++ os-apply-config --key completion-signal --type raw --key-default ''
09:34:41 [2016/09/26 07:29:43 AM] [WARNING] DEPRECATED: falling back to /var/run/os-collect-config/os_config_files.json
09:34:41 + SIGNAL=
09:34:41 ++ os-apply-config --key instance-id --type raw --key-default ''
09:34:42 [2016/09/26 07:29:43 AM] [WARNING] DEPRECATED: falling back to /var/run/os-collect-config/os_config_files.json
09:34:42 + ID=
09:34:42 + '[' -n '' ']'
09:34:42 + exit 0
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 99-refresh-completed completed
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 ----------------------- PROFILING -----------------------
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 Target: post-configure.d
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 Script                                     Seconds
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 ---------------------------------------  ----------
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 10-iptables                                   0.034
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 80-seedstack-masquerade                       0.091
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 98-undercloud-setup                          58.746
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 99-refresh-completed                          0.955
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016
09:34:42 dib-run-parts Mon Sep 26 07:29:43 EDT 2016 --------------------- END PROFILING ---------------------
09:34:42 [2016-09-26 07:29:43,488] (os-refresh-config) [INFO] Completed phase post-configure
09:34:42 os-refresh-config completed successfully
09:34:43 Generated new ssh key in ~/.ssh/id_rsa
09:34:44 Created flavor "baremetal" with profile "None"
09:34:44 Created flavor "control" with profile "control"
09:34:44 Created flavor "compute" with profile "compute"
09:34:44 Created flavor "ceph-storage" with profile "ceph-storage"
09:34:45 Created flavor "block-storage" with profile "block-storage"
09:34:45 Created flavor "swift-storage" with profile "swift-storage"
09:35:46 Timed out waiting for execution 5684d953-0eca-4469-b91e-e2bbeac53f3a to finish. State: RUNNING
09:35:46 Traceback (most recent call last):
09:35:46   File "<string>", line 1, in <module>
09:35:46   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1189, in install
09:35:46     _post_config(instack_env)
09:35:46   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1171, in _post_config
09:35:46     _post_config_mistral(mistral)
09:35:46   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1146, in _post_config_mistral
09:35:46     _create_default_plan(mistral)
09:35:46   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1136, in _create_default_plan
09:35:46     "Timed out creating the default Deployment Plan. Please check "
09:35:46 RuntimeError: Timed out creating the default Deployment Plan. Please check the create_default_deployment_plan execution in Mistral with `openstack workflow execution list`.
09:35:46 Command 'instack-install-undercloud' returned non-zero exit status 1
09:35:46 Failed to deploy undercloud
09:35:46 Build step 'Virtualenv Builder' marked build as failure
09:35:47 Build step 'Groovy Postbuild' marked build as failure
09:35:47 Build step 'Groovy Postbuild' marked build as failure
09:35:47 [BFA] Scanning build for known causes...
09:35:47 [BFA] Found failure cause(s):
09:35:47 [BFA] instack-install-undercloud returned a non-zero status
09:35:47 [BFA] Done. 0s
09:35:47 Started calculate disk usage of build
09:35:47 Finished Calculation of disk usage of build in 0 seconds
09:35:47 Started calculate disk usage of workspace
09:35:47 Finished Calculation of disk usage of workspace in 0 seconds
09:35:47 Finished: FAILURE
Comment 1 Dougal Matthews 2016-10-03 15:05:13 UTC 
This is the same as this upstream bug: https://bugs.launchpad.net/tripleo/+bug/1628388

It was resolved hre: https://github.com/openstack/instack-undercloud/commit/0c86110ef1499db20e816c43c700716417752bdd
Comment 2 James Slagle 2016-10-03 16:13:18 UTC 
ok, thanks for the update. i believe this has been built downstream. can you confirm and if so, set the Fixed in Version and move the bug to MODIFIED?
Comment 4 Alexander Chuzhoy 2016-10-11 19:49:11 UTC 
Verified:

Environment:
instack-undercloud-5.0.0-0.20160930175750.9d2a655.el7ost.noarch


The reported issue doesn't reproduce.
Comment 6 Dougal Matthews 2016-11-01 09:35:45 UTC 
I spotted this error in upstream CI and increased the timeout value. From the logs no error had happened, it just took longer than expected for some reason.

https://review.openstack.org/#/c/391373/
Comment 8 errata-xmlrpc 2016-12-14 16:06:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html