Bug 1249628

Summary:	heat race condition causing deployments to get stuck at various places
Product:	Red Hat OpenStack	Reporter:	Amit Ugol <augol>
Component:	openstack-heat	Assignee:	Steve Baker <sbaker>
Status:	CLOSED ERRATA	QA Contact:	Amit Ugol <augol>
Severity:	high	Docs Contact:
Priority:	high
Version:	Director	CC:	augol, fhubik, gfidente, ggillies, jliberma, jschluet, jstransk, lnatapov, mburns, mcornea, nbarcet, ohochman, opavlenk, rhel-osp-director-maint, rhos-maint, rlandy, sasha, sbaker, shardy, yeylon
Target Milestone:	async	Keywords:	ZStream
Target Release:	7.0 (Kilo)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-heat-2015.1.0-6.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1255759 (view as bug list)		Environment:
Last Closed:	2015-08-17 16:54:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Amit Ugol 2015-08-03 12:42:43 UTC

Description of problem:
Deployments get 'stuck' after a while on various postdeployment_steps.
Since puddle 2015-07-23.1, deploying on various setups would not continue beyond a certain step. The step changes, it is sometimes Compute..._Step4, Controller..._Step7 or any other step.
Its also very hard to debug this issue and up to this point (at least from what I have seen in #rhos-mgt) no-one was able to pinpoint this issue.

Version-Release number of selected component (if applicable):
Since puddle 2015-07-23.1 onward (including latest)

How reproducible:
every once in a while

Steps to Reproduce:
1. From what I could gather, deploy on a virt environment and with a ceph node, but the setups are not the same every time, nor does it matter if we try with/out tuskar and with/out network isolation etc.

Actual results:
Eventually a timeout is reached and the deployment stops. Sometimes re-deploying it will give you a working deployment, sometimes it will be a broken deployment.

Comment 4 Mike Burns 2015-08-03 15:51:08 UTC

This bug is being converted to a tracker.  Please file issues individually in separate bugs and have them block this bug.  Each instance needs to be investigated separately which is not really doable within this one bug.

By itself, this bug is too generic to ever be resolved.

Comment 11 Ola Pavlenko 2015-08-05 08:05:28 UTC

From ohochman:

It seems that the issue reproduce on my setup with the new poodle from 04 Aug 15, 


Deployment command : 
openstack overcloud deploy --plan overcloud --control-scale 1 --compute-scale 1 --ceph-storage-scale 1  --block-storage-scale 1 --swift-storage-scale 1 -e /home/stack/network-environment.yaml --ntp-server 10.5.26.10 --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout  


[stack@rhos-compute-node-18 ~]$ heat resource-list overcloud -n 5 | grep -iv complete
+---------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+----------------------------------------+
| resource_name                               | physical_resource_id                          | resource_type                                     | resource_status    | updated_time         | parent_resource                        |
+---------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+----------------------------------------+
| ComputeNodesPostDeployment                  | c298a324-766a-4a92-91d3-4518c57324bc          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED      | 2015-08-04T23:05:44Z |                                        |
| ControllerNodesPostDeployment               | 7f7607ab-a2a7-4031-98ef-d0ca3ea72bd7          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED      | 2015-08-04T23:05:44Z |                                        |
| ComputePuppetDeployment                     | 9dc6fba3-d54a-4e67-8b5b-af5b6e589cff          | OS::Heat::StructuredDeployments                   | CREATE_IN_PROGRESS | 2015-08-04T23:16:55Z | ComputeNodesPostDeployment             |
| 0                                           | 528200df-3ae9-472e-9818-33d880ebce70          | OS::Heat::StructuredDeployment                    | CREATE_IN_PROGRESS | 2015-08-04T23:16:57Z | ComputePuppetDeployment                |
| ControllerServicesBaseDeployment_Step2      | aee11e78-b9a7-48b9-9caa-2ff1d7bb0b54          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-08-04T23:17:08Z | ControllerNodesPostDeployment          |
| 0                                           | 689316ec-be1e-4387-8518-18883963c74f          | OS::Heat::StructuredDeployment                    | CREATE_FAILED      | 2015-08-04T23:19:01Z | ControllerServicesBaseDeployment_Step2 |
+---------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+----------------------------------------+


controller (journalctl | grep os-collect-config): 
------------ -------------------------------------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: dib-run-parts Tue Aug 4 19:20:41 EDT 2015 99-refresh-completed completed
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: ----------------------- PROFILING -----------------------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: Target: post-configure.d
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: Script                                     Seconds
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: ---------------------------------------  ----------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: 99-refresh-completed                          5.339
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: --------------------- END PROFILING ---------------------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: [2015-08-04 19:20:41,670] (os-refresh-config) [INFO] Completed phase post-configure
Aug 04 19:20:45 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:20:45.414 5953 WARNING os-collect-config [-] Source [request] Unavailable.
Aug 04 19:20:45 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:20:45.415 5953 WARNING os_collect_config.local [-] /var/lib/os-collect-config/local-data not found. Skipping
Aug 04 19:20:45 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:20:45.415 5953 WARNING os_collect_config.local [-] No local metadata found (['/var/lib/os-collect-config/local-data'])
Aug 04 19:21:17 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:21:17.829 5953 WARNING os-collect-config [-] Source [request] Unavailable.


Best Regards,
Omri

Comment 12 Ronelle Landy 2015-08-05 13:09:02 UTC

CI did not hit a stalled deploy in the test runs.
So far so good for CI - with latest poodle fix.

Will continue with runs

Comment 16 Ola Pavlenko 2015-08-05 16:39:53 UTC

Tried to deploy via UI and failed with (I think) the same issue.
Updated the bits  (to the latest poodle) and redeployed successfully.

Comment 17 Amit Ugol 2015-08-06 05:32:17 UTC

(In reply to Ola Pavlenko from comment #16)
> Tried to deploy via UI and failed with (I think) the same issue.
> Updated the bits  (to the latest poodle) and redeployed successfully.

Re-deploying also 'fixes' stuck versions from before the fix...

Comment 18 Amit Ugol 2015-08-06 06:06:33 UTC

Much better now.
If there is still a UI related issue, we need top open one specifically on that.

Comment 24 Leonid Natapov 2015-08-10 14:57:26 UTC

reproduced with openstack-heat-2015.1.0-5.el7ost

Comment 26 Amit Ugol 2015-08-11 04:44:01 UTC

It wouldn't manifest for me but there are still issues around this bug and also a few others have encountered it. Perhaps I OKed it too soon. I am re-opening this but IMHO we can turn the priority down a bit as it is less probable to encounter it in the field.

Comment 33 Omri Hochman 2015-08-14 20:01:35 UTC

Unable to reproduce using : openstack-heat-2015.1.0-5.el7ost.noarch

Verified for now.  will re-open in case of reproduce. 

openstack-heat-api-2015.1.0-5.el7ost.noarch
openstack-heat-common-2015.1.0-5.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch
openstack-heat-engine-2015.1.0-5.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-5.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-5.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch

Comment 36 errata-xmlrpc 2015-08-17 16:54:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1637

Comment 37 Steve Baker 2015-08-20 22:09:59 UTC

Created attachment 1065414 [details]
add logging to diagnose race

The previous fixes should have fixed *a* race but it looks like there may be another issue which is still causing this bug. I have a logging patch which would help with diagnosing where the issue is.

If you have an environment which is affected, please do the following
- apply service_software_config_logging.patch to /usr/lib/python2.7/site-packages/heat/engine/service_software_config.py
- in /etc/heat/heat.conf set the following
  [DEFAULT] debug = False
  [database] connection_debug = 50
- truncate --size 0 /var/log/heat/heat-engine.log
- systemctl restart openstack-heat-engine
- deploy overcloud until problem recurs
- Make /var/log/heat/heat-engine.log available as a comment in this bug

Comment 38 Graeme Gillies 2015-08-21 06:49:57 UTC

Created attachment 1065448 [details]
heat-engine.log with failure in step5

Hi,

Reproduced problem with the patch from Comment 37 though unfortunately I had sql debugging turned off.

It failed at Step5

Attached is log

I'll try to reproduce with sql logging turned on as well and attach

Regards,

Graeme

Comment 39 Graeme Gillies 2015-08-21 06:50:54 UTC

output of my stack list as well to help match to log

$ heat stack-list --show-nested
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+
| id                                   | stack_name                                                                                                    | stack_status       | creation_time        | parent                               |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+
| 1d0744f2-a3db-4f1f-b68e-7281352e7a17 | overcloud                                                                                                     | CREATE_IN_PROGRESS | 2015-08-21T05:57:47Z | None                                 |
| 37b9045a-67f9-4def-b97a-1a8677f91fbd | overcloud-Networks-iuunei3g4d7i                                                                               | CREATE_COMPLETE    | 2015-08-21T05:57:50Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 748c743f-6f0a-4bc0-9306-6c9a64b01060 | overcloud-Networks-iuunei3g4d7i-StorageNetwork-qwz2kqao2xqe                                                   | CREATE_COMPLETE    | 2015-08-21T05:57:51Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| da23de84-21ae-4e5d-930a-3a9ea9589885 | overcloud-VipConfig-56czet7rcdlv                                                                              | CREATE_COMPLETE    | 2015-08-21T05:57:51Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| c749e566-dd81-4830-bf2b-a399d5400ac6 | overcloud-Networks-iuunei3g4d7i-StorageMgmtNetwork-forc3jbtk7vw                                               | CREATE_COMPLETE    | 2015-08-21T05:57:52Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| f70b8abd-e6a3-4693-8928-5043377cac69 | overcloud-Networks-iuunei3g4d7i-InternalNetwork-lo4zyz2i5uiw                                                  | CREATE_COMPLETE    | 2015-08-21T05:57:53Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| e947cff1-6b66-443c-a809-0d5432e83f1e | overcloud-Networks-iuunei3g4d7i-ExternalNetwork-qpnukszxw5tc                                                  | CREATE_COMPLETE    | 2015-08-21T05:57:54Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| e22fddb9-be88-49f3-aa1e-7edb4dd5d8aa | overcloud-Networks-iuunei3g4d7i-TenantNetwork-a5wucf74h5bg                                                    | CREATE_COMPLETE    | 2015-08-21T05:57:56Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| 4a125772-6032-4f04-b163-0b9a9020620e | overcloud-ObjectStorage-v5nm44zukdrb                                                                          | CREATE_COMPLETE    | 2015-08-21T05:58:02Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 987f51eb-ed3f-45ec-af6c-febf047488a3 | overcloud-CephStorage-72trllu54fed                                                                            | CREATE_COMPLETE    | 2015-08-21T05:58:04Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4065b9ea-1182-46d6-b8f5-0e84e8189859 | overcloud-RedisVirtualIP-fldb36xjm7b2                                                                         | CREATE_COMPLETE    | 2015-08-21T05:58:08Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| da53a9c0-27f8-40cd-997b-dc827abaeef1 | overcloud-StorageMgmtVirtualIP-ossak4ifhwv7                                                                   | CREATE_COMPLETE    | 2015-08-21T05:58:09Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 16ac36a4-b922-4ede-89bd-32dcccfa7f4f | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh                                                             | CREATE_COMPLETE    | 2015-08-21T05:58:10Z | 987f51eb-ed3f-45ec-af6c-febf047488a3 |
| 8349805b-c043-4b1b-b388-40c46a527e2d | overcloud-PublicVirtualIP-3nbhyn4viqol                                                                        | CREATE_COMPLETE    | 2015-08-21T05:58:11Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f301cd61-a8d8-41cc-ba84-4281541e2b15 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-UpdateConfig-shtbycdgyses                                   | CREATE_COMPLETE    | 2015-08-21T05:58:11Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 6d54e85f-35d2-4f54-9dbb-769f04ada11b | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NodeUserData-ien5lw6zjykw                                   | CREATE_COMPLETE    | 2015-08-21T05:58:12Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| cd9bb970-b0c8-49d9-a564-7de3fb56a517 | overcloud-StorageVirtualIP-pumvnrdn2etp                                                                       | CREATE_COMPLETE    | 2015-08-21T05:58:13Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 252a8809-ce00-4a9d-8088-891447efad55 | overcloud-InternalApiVirtualIP-c4c5qezsqfwn                                                                   | CREATE_COMPLETE    | 2015-08-21T05:58:15Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f81bcdf3-21d6-41f0-84eb-6d529af4b6da | overcloud-VipMap-van73sfst7kt                                                                                 | CREATE_COMPLETE    | 2015-08-21T05:58:20Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 3be18f3d-28a4-4b7f-aa49-3d1f55f61dcc | overcloud-Compute-gp2hrbniskdh                                                                                | CREATE_COMPLETE    | 2015-08-21T05:58:25Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 22d8dd30-012f-4fe9-8c53-01a69193bd4b | overcloud-Controller-5rcadg5ssquk                                                                             | CREATE_COMPLETE    | 2015-08-21T05:58:31Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah                                                                 | CREATE_COMPLETE    | 2015-08-21T05:58:34Z | 3be18f3d-28a4-4b7f-aa49-3d1f55f61dcc |
| 1cb132da-ee9f-4488-8865-6864657e557f | overcloud-BlockStorage-wgi7df3xot25                                                                           | CREATE_COMPLETE    | 2015-08-21T05:58:36Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 5666450e-3a1e-4e95-9baf-414b27640622 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-UpdateConfig-zgy3wxxkxzwj                                       | CREATE_COMPLETE    | 2015-08-21T05:58:38Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 7b744a27-144e-4ab1-990a-2ab18cd8f669 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NodeUserData-2srxjq24dzli                                       | CREATE_COMPLETE    | 2015-08-21T05:58:40Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 4a93ae08-58c5-4704-bf56-a260e483b61e | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u                                                              | CREATE_COMPLETE    | 2015-08-21T05:58:49Z | 22d8dd30-012f-4fe9-8c53-01a69193bd4b |
| 72dc0ac8-53ba-4778-bfe4-b758aae9b933 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NodeUserData-dojuhv37dei6                                    | CREATE_COMPLETE    | 2015-08-21T05:58:56Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 23eb057e-6597-4857-85a4-9d9db9e280b1 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-UpdateConfig-d3rvqgbwol3o                                    | CREATE_COMPLETE    | 2015-08-21T05:59:01Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 65b9ad48-a2b6-4e20-b8f2-404390a03b82 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-StorageMgmtPort-uugaoc3sppyj                                | CREATE_COMPLETE    | 2015-08-21T06:17:05Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 3a103c70-8719-4a03-b5a2-e50236d08eec | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-StoragePort-dizfwzvwskyk                                    | CREATE_COMPLETE    | 2015-08-21T06:17:15Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 95bf01c8-f4b9-4738-b21a-c98dc54c592f | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetworkConfig-y4ghztkohfh7                                  | CREATE_COMPLETE    | 2015-08-21T06:17:25Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| f4d35c64-3916-4ff3-bdb4-ed347f58ccf0 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetIpMap-ium4vgx4sd53                                       | CREATE_COMPLETE    | 2015-08-21T06:17:33Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 2564b862-a1be-4a2a-92ae-96cd9be9fc0d | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetIpSubnetMap-wi5pko7qsa6l                                 | CREATE_COMPLETE    | 2015-08-21T06:17:35Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 83d16435-f9ee-46c8-8569-a914ab2966c9 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-TenantPort-v7j5vfl5wrd2                                         | CREATE_COMPLETE    | 2015-08-21T06:17:58Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| a1cfe06c-bac4-4dc5-9663-012661451c8b | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-StoragePort-lnvmgslwglng                                        | CREATE_COMPLETE    | 2015-08-21T06:18:01Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| fd832499-736d-4905-97ec-85efcbd367bd | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-InternalApiPort-23jfu3zmefwv                                    | CREATE_COMPLETE    | 2015-08-21T06:18:05Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 5df6f8a6-b104-4eca-b155-5982ea15acd7 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NetIpMap-v6ny5zct6f7s                                           | CREATE_COMPLETE    | 2015-08-21T06:18:11Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| e1d62c12-4f03-421b-b22f-8fa127be8060 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NetworkConfig-msw4lwut5z7g                                      | CREATE_COMPLETE    | 2015-08-21T06:18:13Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 339d5aad-70f7-42c3-afa3-deef4f6def71 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-TenantPort-4bw5djiof7n5                                      | CREATE_COMPLETE    | 2015-08-21T06:18:14Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| a2dab192-11ba-48f9-873b-bc45a6304e82 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-ExternalPort-fjoqr42svs3h                                    | CREATE_COMPLETE    | 2015-08-21T06:18:18Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 7017e935-fc85-42a1-9b40-520b15a434a7 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-StorageMgmtPort-xn4jeyizhas3                                 | CREATE_COMPLETE    | 2015-08-21T06:18:22Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 708eb94d-83f1-41d4-82f7-a95132c6ff96 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-InternalApiPort-nfw465dpjkiw                                 | CREATE_COMPLETE    | 2015-08-21T06:18:27Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 2e05687d-6200-47be-8b3f-95b52b524a45 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-StoragePort-fn3ohp4oflka                                     | CREATE_COMPLETE    | 2015-08-21T06:18:30Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| b52572cc-5c57-4c6f-b182-25f858557966 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetIpSubnetMap-6a6cjlotsect                                  | CREATE_COMPLETE    | 2015-08-21T06:18:41Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 561a6ec7-9b47-41bb-a291-518a4235500e | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetworkConfig-o6hvbchm4egu                                   | CREATE_COMPLETE    | 2015-08-21T06:18:44Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 62500da2-e04a-4b12-9019-12a0e4c3c9ec | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetIpMap-mhcjq6slbief                                        | CREATE_COMPLETE    | 2015-08-21T06:18:47Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 48b6805d-5c94-4d1c-8d63-e48f17970898 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-ControllerExtraConfigPre-bwysoleuh7bg                        | CREATE_COMPLETE    | 2015-08-21T06:22:31Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| d7972546-81ec-4862-a456-abfec868297e | overcloud-ControllerBootstrapNodeConfig-ll3enwdxdhcr                                                          | CREATE_COMPLETE    | 2015-08-21T06:22:36Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 68e6ae96-37ac-445b-bdb4-fe0fa023fd9c | overcloud-ControllerIpListMap-ta4r45cszs2r                                                                    | CREATE_COMPLETE    | 2015-08-21T06:22:39Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| ac918a51-f06a-4e5c-94e2-2b252db30008 | overcloud-VipDeployment-dnbfv45aypxc                                                                          | CREATE_COMPLETE    | 2015-08-21T06:22:42Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| d5cf377f-0acc-420e-b8ae-fc8d82c10003 | overcloud-SwiftDevicesAndProxyConfig-2oid4zqloy3j                                                             | CREATE_COMPLETE    | 2015-08-21T06:22:44Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 462a4a30-184f-49b1-b15b-77525f6b5017 | overcloud-ControllerBootstrapNodeDeployment-5j56m5ayrzem                                                      | CREATE_COMPLETE    | 2015-08-21T06:22:48Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| bc6cb3d0-849e-4c38-856b-26113bf8320e | overcloud-ControllerClusterDeployment-g4wr453gjmg3                                                            | CREATE_COMPLETE    | 2015-08-21T06:22:50Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 03b6eb84-4115-4ab1-9a38-15ce432068b3 | overcloud-ControllerSwiftDeployment-zhzes7y6kgmn                                                              | CREATE_COMPLETE    | 2015-08-21T06:22:53Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| fe6fe74e-cba8-48bb-a8f6-7703b326d627 | overcloud-ObjectStorageSwiftDeployment-aumsowahccrx                                                           | CREATE_COMPLETE    | 2015-08-21T06:22:55Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4e16b903-1664-4695-b54d-db690bb9e76b | overcloud-allNodesConfig-3izzrpt4oava                                                                         | CREATE_COMPLETE    | 2015-08-21T06:22:58Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 653149ab-11f3-4481-bad7-cf369d04795c | overcloud-CephClusterConfig-dj3p7hnjotcl                                                                      | CREATE_COMPLETE    | 2015-08-21T06:22:59Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 8c2b6239-5b1d-41a6-b0ae-df36bac5d5f0 | overcloud-CephStorageAllNodesDeployment-4goz26gg6vut                                                          | CREATE_COMPLETE    | 2015-08-21T06:23:06Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| ac8ae8a9-2a22-4735-861b-0660c60b044f | overcloud-ControllerCephDeployment-adxqq3scn3wz                                                               | CREATE_COMPLETE    | 2015-08-21T06:23:12Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4f0449ff-c3fe-4af1-907a-15fd35f4447e | overcloud-ControllerAllNodesDeployment-kpfs37qjm7n6                                                           | CREATE_COMPLETE    | 2015-08-21T06:23:17Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 734ae360-5720-4248-96de-fcc53ab24a12 | overcloud-ComputeCephDeployment-mlqwqifsrfrb                                                                  | CREATE_COMPLETE    | 2015-08-21T06:23:19Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 271aa9ef-a118-48eb-b4d1-92c20538d38e | overcloud-CephStorageCephDeployment-6zw6ajmb3qpp                                                              | CREATE_COMPLETE    | 2015-08-21T06:23:23Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 0a5a1ce9-7a59-4dd3-a501-4e4859d0fb23 | overcloud-ObjectStorageAllNodesDeployment-htmvldloqm3f                                                        | CREATE_COMPLETE    | 2015-08-21T06:23:25Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 6a559898-0cdc-4337-835d-601b41b9cb1c | overcloud-ComputeAllNodesDeployment-mgrucunikgpb                                                              | CREATE_COMPLETE    | 2015-08-21T06:23:28Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f5ac100a-05c1-4994-973b-4fdbbe45ec9d | overcloud-BlockStorageAllNodesDeployment-pvgrw2c2dnf6                                                         | CREATE_COMPLETE    | 2015-08-21T06:23:31Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 10d908a7-82ec-41c5-b7b7-40278b72f759 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e                                                       | CREATE_COMPLETE    | 2015-08-21T06:23:37Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 55c0b272-4c85-4318-a816-341f72f36899 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-StorageDeployment_Step1-ayikgu7mbjjy                  | CREATE_COMPLETE    | 2015-08-21T06:23:40Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| 6568fbe9-22fd-4d3a-b273-351bd1c0f180 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-StorageRingbuilderDeployment_Step2-g2bfxw743vyl       | CREATE_COMPLETE    | 2015-08-21T06:23:44Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| ccf75bf8-825a-4d79-9a3a-3abf403dfe84 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-ExtraConfig-grkpa5cyaj4j                              | CREATE_COMPLETE    | 2015-08-21T06:23:49Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| 10b8ea8d-89b7-47b8-8db4-59783222d2c8 | overcloud-ComputeNodesPostDeployment-sj3pzjzcnexa                                                             | CREATE_IN_PROGRESS | 2015-08-21T06:24:34Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f5125554-08c1-4e3f-a5c0-4efcb20a03fb | overcloud-ComputeNodesPostDeployment-sj3pzjzcnexa-ComputePuppetDeployment-jmdrg76fgnkk                        | CREATE_IN_PROGRESS | 2015-08-21T06:24:37Z | 10b8ea8d-89b7-47b8-8db4-59783222d2c8 |
| 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs                                                          | CREATE_IN_PROGRESS | 2015-08-21T06:25:17Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| caf1fca4-953d-4cde-b257-83be74837557 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerPuppetConfig-5a6vhoyawz7n                      | CREATE_COMPLETE    | 2015-08-21T06:25:19Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 8aefd1d4-3b0f-4db2-bd82-b45d256a22f2 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerLoadBalancerDeployment_Step1-wx347abkxfah      | CREATE_COMPLETE    | 2015-08-21T06:25:24Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 0796df35-716c-40e0-b52b-35cb157bca82 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerServicesBaseDeployment_Step2-uwysertxhbs5      | CREATE_COMPLETE    | 2015-08-21T06:28:17Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| e28483ee-58fc-4fcf-b83b-32fb82273849 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerRingbuilderDeployment_Step3-crgoagoipcqu       | CREATE_COMPLETE    | 2015-08-21T06:31:31Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 5b35e391-34dc-4c51-9ba0-68cab3be97c4 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerOvercloudServicesDeployment_Step4-7dbmd7npeax4 | CREATE_COMPLETE    | 2015-08-21T06:33:03Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 5c97b3ed-7ef0-4b4a-a977-be30914fe4f5 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerOvercloudServicesDeployment_Step5-juj37e7skuvx | CREATE_IN_PROGRESS | 2015-08-21T06:40:47Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+

Comment 40 Mike Burns 2015-08-21 13:32:54 UTC

This bug is closed.  I've cloned it to a separate bug (Bug 1255759).  Please make any further updates there and not in this bug.