1249628 – heat race condition causing deployments to get stuck at various places

Bug 1249628 - heat race condition causing deployments to get stuck at various places

Summary: heat race condition causing deployments to get stuck at various places

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-heat
Sub Component:
Version:	Director
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	async
Target Release:	7.0 (Kilo)
Assignee:	Steve Baker
QA Contact:	Amit Ugol
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-08-03 12:42 UTC by Amit Ugol
Modified:	2023-02-22 23:02 UTC (History)
CC List:	20 users (show)
Fixed In Version:	openstack-heat-2015.1.0-6.el7ost
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1255759 (view as bug list)
Environment:
Last Closed:	2015-08-17 16:54:50 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	204780	0	None	MERGED	Always commit after deployment db operations	2020-08-25 13:06:33 UTC
OpenStack gerrit	208777	0	None	MERGED	Always commit after deployment db operations	2020-08-25 13:06:33 UTC
OpenStack gerrit	208778	0	None	MERGED	Use atomic_key for deployment metadata updates	2020-08-25 13:06:33 UTC
Red Hat Bugzilla	1264236	1	None	None	None	2021-01-20 06:05:38 UTC
Red Hat Product Errata	RHBA-2015:1637	0	normal	SHIPPED_LIVE	openstack-heat bug fix advisory	2015-08-17 20:54:38 UTC

Internal Links: 1264236

Description Amit Ugol 2015-08-03 12:42:43 UTC

Description of problem:
Deployments get 'stuck' after a while on various postdeployment_steps.
Since puddle 2015-07-23.1, deploying on various setups would not continue beyond a certain step. The step changes, it is sometimes Compute..._Step4, Controller..._Step7 or any other step.
Its also very hard to debug this issue and up to this point (at least from what I have seen in #rhos-mgt) no-one was able to pinpoint this issue.

Version-Release number of selected component (if applicable):
Since puddle 2015-07-23.1 onward (including latest)

How reproducible:
every once in a while

Steps to Reproduce:
1. From what I could gather, deploy on a virt environment and with a ceph node, but the setups are not the same every time, nor does it matter if we try with/out tuskar and with/out network isolation etc.

Actual results:
Eventually a timeout is reached and the deployment stops. Sometimes re-deploying it will give you a working deployment, sometimes it will be a broken deployment.

Comment 4 Mike Burns 2015-08-03 15:51:08 UTC

This bug is being converted to a tracker.  Please file issues individually in separate bugs and have them block this bug.  Each instance needs to be investigated separately which is not really doable within this one bug.

By itself, this bug is too generic to ever be resolved.

Comment 11 Ola Pavlenko 2015-08-05 08:05:28 UTC

From ohochman:

It seems that the issue reproduce on my setup with the new poodle from 04 Aug 15, 


Deployment command : 
openstack overcloud deploy --plan overcloud --control-scale 1 --compute-scale 1 --ceph-storage-scale 1  --block-storage-scale 1 --swift-storage-scale 1 -e /home/stack/network-environment.yaml --ntp-server 10.5.26.10 --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout  


[stack@rhos-compute-node-18 ~]$ heat resource-list overcloud -n 5 | grep -iv complete
+---------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+----------------------------------------+
| resource_name                               | physical_resource_id                          | resource_type                                     | resource_status    | updated_time         | parent_resource                        |
+---------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+----------------------------------------+
| ComputeNodesPostDeployment                  | c298a324-766a-4a92-91d3-4518c57324bc          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED      | 2015-08-04T23:05:44Z |                                        |
| ControllerNodesPostDeployment               | 7f7607ab-a2a7-4031-98ef-d0ca3ea72bd7          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED      | 2015-08-04T23:05:44Z |                                        |
| ComputePuppetDeployment                     | 9dc6fba3-d54a-4e67-8b5b-af5b6e589cff          | OS::Heat::StructuredDeployments                   | CREATE_IN_PROGRESS | 2015-08-04T23:16:55Z | ComputeNodesPostDeployment             |
| 0                                           | 528200df-3ae9-472e-9818-33d880ebce70          | OS::Heat::StructuredDeployment                    | CREATE_IN_PROGRESS | 2015-08-04T23:16:57Z | ComputePuppetDeployment                |
| ControllerServicesBaseDeployment_Step2      | aee11e78-b9a7-48b9-9caa-2ff1d7bb0b54          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-08-04T23:17:08Z | ControllerNodesPostDeployment          |
| 0                                           | 689316ec-be1e-4387-8518-18883963c74f          | OS::Heat::StructuredDeployment                    | CREATE_FAILED      | 2015-08-04T23:19:01Z | ControllerServicesBaseDeployment_Step2 |
+---------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+----------------------+----------------------------------------+


controller (journalctl | grep os-collect-config): 
------------ -------------------------------------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: dib-run-parts Tue Aug 4 19:20:41 EDT 2015 99-refresh-completed completed
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: ----------------------- PROFILING -----------------------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: Target: post-configure.d
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: Script                                     Seconds
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: ---------------------------------------  ----------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: 99-refresh-completed                          5.339
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: --------------------- END PROFILING ---------------------
Aug 04 19:20:41 overcloud-controller-0.localdomain os-collect-config[5953]: [2015-08-04 19:20:41,670] (os-refresh-config) [INFO] Completed phase post-configure
Aug 04 19:20:45 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:20:45.414 5953 WARNING os-collect-config [-] Source [request] Unavailable.
Aug 04 19:20:45 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:20:45.415 5953 WARNING os_collect_config.local [-] /var/lib/os-collect-config/local-data not found. Skipping
Aug 04 19:20:45 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:20:45.415 5953 WARNING os_collect_config.local [-] No local metadata found (['/var/lib/os-collect-config/local-data'])
Aug 04 19:21:17 overcloud-controller-0.localdomain os-collect-config[5953]: 2015-08-04 19:21:17.829 5953 WARNING os-collect-config [-] Source [request] Unavailable.


Best Regards,
Omri

Comment 12 Ronelle Landy 2015-08-05 13:09:02 UTC

CI did not hit a stalled deploy in the test runs.
So far so good for CI - with latest poodle fix.

Will continue with runs

Comment 16 Ola Pavlenko 2015-08-05 16:39:53 UTC

Tried to deploy via UI and failed with (I think) the same issue.
Updated the bits  (to the latest poodle) and redeployed successfully.

Comment 17 Amit Ugol 2015-08-06 05:32:17 UTC

(In reply to Ola Pavlenko from comment #16)
> Tried to deploy via UI and failed with (I think) the same issue.
> Updated the bits  (to the latest poodle) and redeployed successfully.

Re-deploying also 'fixes' stuck versions from before the fix...

Comment 18 Amit Ugol 2015-08-06 06:06:33 UTC

Much better now.
If there is still a UI related issue, we need top open one specifically on that.

Comment 24 Leonid Natapov 2015-08-10 14:57:26 UTC

reproduced with openstack-heat-2015.1.0-5.el7ost

Comment 26 Amit Ugol 2015-08-11 04:44:01 UTC

It wouldn't manifest for me but there are still issues around this bug and also a few others have encountered it. Perhaps I OKed it too soon. I am re-opening this but IMHO we can turn the priority down a bit as it is less probable to encounter it in the field.

Comment 33 Omri Hochman 2015-08-14 20:01:35 UTC

Unable to reproduce using : openstack-heat-2015.1.0-5.el7ost.noarch

Verified for now.  will re-open in case of reproduce. 

openstack-heat-api-2015.1.0-5.el7ost.noarch
openstack-heat-common-2015.1.0-5.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-46.el7ost.noarch
openstack-heat-engine-2015.1.0-5.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-5.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-5.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch

Comment 36 errata-xmlrpc 2015-08-17 16:54:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1637

Comment 37 Steve Baker 2015-08-20 22:09:59 UTC

Created attachment 1065414 [details]
add logging to diagnose race

The previous fixes should have fixed *a* race but it looks like there may be another issue which is still causing this bug. I have a logging patch which would help with diagnosing where the issue is.

If you have an environment which is affected, please do the following
- apply service_software_config_logging.patch to /usr/lib/python2.7/site-packages/heat/engine/service_software_config.py
- in /etc/heat/heat.conf set the following
  [DEFAULT] debug = False
  [database] connection_debug = 50
- truncate --size 0 /var/log/heat/heat-engine.log
- systemctl restart openstack-heat-engine
- deploy overcloud until problem recurs
- Make /var/log/heat/heat-engine.log available as a comment in this bug

Comment 38 Graeme Gillies 2015-08-21 06:49:57 UTC

Created attachment 1065448 [details]
heat-engine.log with failure in step5

Hi,

Reproduced problem with the patch from Comment 37 though unfortunately I had sql debugging turned off.

It failed at Step5

Attached is log

I'll try to reproduce with sql logging turned on as well and attach

Regards,

Graeme

Comment 39 Graeme Gillies 2015-08-21 06:50:54 UTC

output of my stack list as well to help match to log

$ heat stack-list --show-nested
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+
| id                                   | stack_name                                                                                                    | stack_status       | creation_time        | parent                               |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+
| 1d0744f2-a3db-4f1f-b68e-7281352e7a17 | overcloud                                                                                                     | CREATE_IN_PROGRESS | 2015-08-21T05:57:47Z | None                                 |
| 37b9045a-67f9-4def-b97a-1a8677f91fbd | overcloud-Networks-iuunei3g4d7i                                                                               | CREATE_COMPLETE    | 2015-08-21T05:57:50Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 748c743f-6f0a-4bc0-9306-6c9a64b01060 | overcloud-Networks-iuunei3g4d7i-StorageNetwork-qwz2kqao2xqe                                                   | CREATE_COMPLETE    | 2015-08-21T05:57:51Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| da23de84-21ae-4e5d-930a-3a9ea9589885 | overcloud-VipConfig-56czet7rcdlv                                                                              | CREATE_COMPLETE    | 2015-08-21T05:57:51Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| c749e566-dd81-4830-bf2b-a399d5400ac6 | overcloud-Networks-iuunei3g4d7i-StorageMgmtNetwork-forc3jbtk7vw                                               | CREATE_COMPLETE    | 2015-08-21T05:57:52Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| f70b8abd-e6a3-4693-8928-5043377cac69 | overcloud-Networks-iuunei3g4d7i-InternalNetwork-lo4zyz2i5uiw                                                  | CREATE_COMPLETE    | 2015-08-21T05:57:53Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| e947cff1-6b66-443c-a809-0d5432e83f1e | overcloud-Networks-iuunei3g4d7i-ExternalNetwork-qpnukszxw5tc                                                  | CREATE_COMPLETE    | 2015-08-21T05:57:54Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| e22fddb9-be88-49f3-aa1e-7edb4dd5d8aa | overcloud-Networks-iuunei3g4d7i-TenantNetwork-a5wucf74h5bg                                                    | CREATE_COMPLETE    | 2015-08-21T05:57:56Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| 4a125772-6032-4f04-b163-0b9a9020620e | overcloud-ObjectStorage-v5nm44zukdrb                                                                          | CREATE_COMPLETE    | 2015-08-21T05:58:02Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 987f51eb-ed3f-45ec-af6c-febf047488a3 | overcloud-CephStorage-72trllu54fed                                                                            | CREATE_COMPLETE    | 2015-08-21T05:58:04Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4065b9ea-1182-46d6-b8f5-0e84e8189859 | overcloud-RedisVirtualIP-fldb36xjm7b2                                                                         | CREATE_COMPLETE    | 2015-08-21T05:58:08Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| da53a9c0-27f8-40cd-997b-dc827abaeef1 | overcloud-StorageMgmtVirtualIP-ossak4ifhwv7                                                                   | CREATE_COMPLETE    | 2015-08-21T05:58:09Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 16ac36a4-b922-4ede-89bd-32dcccfa7f4f | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh                                                             | CREATE_COMPLETE    | 2015-08-21T05:58:10Z | 987f51eb-ed3f-45ec-af6c-febf047488a3 |
| 8349805b-c043-4b1b-b388-40c46a527e2d | overcloud-PublicVirtualIP-3nbhyn4viqol                                                                        | CREATE_COMPLETE    | 2015-08-21T05:58:11Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f301cd61-a8d8-41cc-ba84-4281541e2b15 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-UpdateConfig-shtbycdgyses                                   | CREATE_COMPLETE    | 2015-08-21T05:58:11Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 6d54e85f-35d2-4f54-9dbb-769f04ada11b | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NodeUserData-ien5lw6zjykw                                   | CREATE_COMPLETE    | 2015-08-21T05:58:12Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| cd9bb970-b0c8-49d9-a564-7de3fb56a517 | overcloud-StorageVirtualIP-pumvnrdn2etp                                                                       | CREATE_COMPLETE    | 2015-08-21T05:58:13Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 252a8809-ce00-4a9d-8088-891447efad55 | overcloud-InternalApiVirtualIP-c4c5qezsqfwn                                                                   | CREATE_COMPLETE    | 2015-08-21T05:58:15Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f81bcdf3-21d6-41f0-84eb-6d529af4b6da | overcloud-VipMap-van73sfst7kt                                                                                 | CREATE_COMPLETE    | 2015-08-21T05:58:20Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 3be18f3d-28a4-4b7f-aa49-3d1f55f61dcc | overcloud-Compute-gp2hrbniskdh                                                                                | CREATE_COMPLETE    | 2015-08-21T05:58:25Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 22d8dd30-012f-4fe9-8c53-01a69193bd4b | overcloud-Controller-5rcadg5ssquk                                                                             | CREATE_COMPLETE    | 2015-08-21T05:58:31Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah                                                                 | CREATE_COMPLETE    | 2015-08-21T05:58:34Z | 3be18f3d-28a4-4b7f-aa49-3d1f55f61dcc |
| 1cb132da-ee9f-4488-8865-6864657e557f | overcloud-BlockStorage-wgi7df3xot25                                                                           | CREATE_COMPLETE    | 2015-08-21T05:58:36Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 5666450e-3a1e-4e95-9baf-414b27640622 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-UpdateConfig-zgy3wxxkxzwj                                       | CREATE_COMPLETE    | 2015-08-21T05:58:38Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 7b744a27-144e-4ab1-990a-2ab18cd8f669 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NodeUserData-2srxjq24dzli                                       | CREATE_COMPLETE    | 2015-08-21T05:58:40Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 4a93ae08-58c5-4704-bf56-a260e483b61e | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u                                                              | CREATE_COMPLETE    | 2015-08-21T05:58:49Z | 22d8dd30-012f-4fe9-8c53-01a69193bd4b |
| 72dc0ac8-53ba-4778-bfe4-b758aae9b933 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NodeUserData-dojuhv37dei6                                    | CREATE_COMPLETE    | 2015-08-21T05:58:56Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 23eb057e-6597-4857-85a4-9d9db9e280b1 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-UpdateConfig-d3rvqgbwol3o                                    | CREATE_COMPLETE    | 2015-08-21T05:59:01Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 65b9ad48-a2b6-4e20-b8f2-404390a03b82 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-StorageMgmtPort-uugaoc3sppyj                                | CREATE_COMPLETE    | 2015-08-21T06:17:05Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 3a103c70-8719-4a03-b5a2-e50236d08eec | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-StoragePort-dizfwzvwskyk                                    | CREATE_COMPLETE    | 2015-08-21T06:17:15Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 95bf01c8-f4b9-4738-b21a-c98dc54c592f | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetworkConfig-y4ghztkohfh7                                  | CREATE_COMPLETE    | 2015-08-21T06:17:25Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| f4d35c64-3916-4ff3-bdb4-ed347f58ccf0 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetIpMap-ium4vgx4sd53                                       | CREATE_COMPLETE    | 2015-08-21T06:17:33Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 2564b862-a1be-4a2a-92ae-96cd9be9fc0d | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetIpSubnetMap-wi5pko7qsa6l                                 | CREATE_COMPLETE    | 2015-08-21T06:17:35Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 83d16435-f9ee-46c8-8569-a914ab2966c9 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-TenantPort-v7j5vfl5wrd2                                         | CREATE_COMPLETE    | 2015-08-21T06:17:58Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| a1cfe06c-bac4-4dc5-9663-012661451c8b | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-StoragePort-lnvmgslwglng                                        | CREATE_COMPLETE    | 2015-08-21T06:18:01Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| fd832499-736d-4905-97ec-85efcbd367bd | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-InternalApiPort-23jfu3zmefwv                                    | CREATE_COMPLETE    | 2015-08-21T06:18:05Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 5df6f8a6-b104-4eca-b155-5982ea15acd7 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NetIpMap-v6ny5zct6f7s                                           | CREATE_COMPLETE    | 2015-08-21T06:18:11Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| e1d62c12-4f03-421b-b22f-8fa127be8060 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NetworkConfig-msw4lwut5z7g                                      | CREATE_COMPLETE    | 2015-08-21T06:18:13Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 339d5aad-70f7-42c3-afa3-deef4f6def71 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-TenantPort-4bw5djiof7n5                                      | CREATE_COMPLETE    | 2015-08-21T06:18:14Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| a2dab192-11ba-48f9-873b-bc45a6304e82 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-ExternalPort-fjoqr42svs3h                                    | CREATE_COMPLETE    | 2015-08-21T06:18:18Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 7017e935-fc85-42a1-9b40-520b15a434a7 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-StorageMgmtPort-xn4jeyizhas3                                 | CREATE_COMPLETE    | 2015-08-21T06:18:22Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 708eb94d-83f1-41d4-82f7-a95132c6ff96 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-InternalApiPort-nfw465dpjkiw                                 | CREATE_COMPLETE    | 2015-08-21T06:18:27Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 2e05687d-6200-47be-8b3f-95b52b524a45 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-StoragePort-fn3ohp4oflka                                     | CREATE_COMPLETE    | 2015-08-21T06:18:30Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| b52572cc-5c57-4c6f-b182-25f858557966 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetIpSubnetMap-6a6cjlotsect                                  | CREATE_COMPLETE    | 2015-08-21T06:18:41Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 561a6ec7-9b47-41bb-a291-518a4235500e | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetworkConfig-o6hvbchm4egu                                   | CREATE_COMPLETE    | 2015-08-21T06:18:44Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 62500da2-e04a-4b12-9019-12a0e4c3c9ec | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetIpMap-mhcjq6slbief                                        | CREATE_COMPLETE    | 2015-08-21T06:18:47Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 48b6805d-5c94-4d1c-8d63-e48f17970898 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-ControllerExtraConfigPre-bwysoleuh7bg                        | CREATE_COMPLETE    | 2015-08-21T06:22:31Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| d7972546-81ec-4862-a456-abfec868297e | overcloud-ControllerBootstrapNodeConfig-ll3enwdxdhcr                                                          | CREATE_COMPLETE    | 2015-08-21T06:22:36Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 68e6ae96-37ac-445b-bdb4-fe0fa023fd9c | overcloud-ControllerIpListMap-ta4r45cszs2r                                                                    | CREATE_COMPLETE    | 2015-08-21T06:22:39Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| ac918a51-f06a-4e5c-94e2-2b252db30008 | overcloud-VipDeployment-dnbfv45aypxc                                                                          | CREATE_COMPLETE    | 2015-08-21T06:22:42Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| d5cf377f-0acc-420e-b8ae-fc8d82c10003 | overcloud-SwiftDevicesAndProxyConfig-2oid4zqloy3j                                                             | CREATE_COMPLETE    | 2015-08-21T06:22:44Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 462a4a30-184f-49b1-b15b-77525f6b5017 | overcloud-ControllerBootstrapNodeDeployment-5j56m5ayrzem                                                      | CREATE_COMPLETE    | 2015-08-21T06:22:48Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| bc6cb3d0-849e-4c38-856b-26113bf8320e | overcloud-ControllerClusterDeployment-g4wr453gjmg3                                                            | CREATE_COMPLETE    | 2015-08-21T06:22:50Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 03b6eb84-4115-4ab1-9a38-15ce432068b3 | overcloud-ControllerSwiftDeployment-zhzes7y6kgmn                                                              | CREATE_COMPLETE    | 2015-08-21T06:22:53Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| fe6fe74e-cba8-48bb-a8f6-7703b326d627 | overcloud-ObjectStorageSwiftDeployment-aumsowahccrx                                                           | CREATE_COMPLETE    | 2015-08-21T06:22:55Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4e16b903-1664-4695-b54d-db690bb9e76b | overcloud-allNodesConfig-3izzrpt4oava                                                                         | CREATE_COMPLETE    | 2015-08-21T06:22:58Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 653149ab-11f3-4481-bad7-cf369d04795c | overcloud-CephClusterConfig-dj3p7hnjotcl                                                                      | CREATE_COMPLETE    | 2015-08-21T06:22:59Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 8c2b6239-5b1d-41a6-b0ae-df36bac5d5f0 | overcloud-CephStorageAllNodesDeployment-4goz26gg6vut                                                          | CREATE_COMPLETE    | 2015-08-21T06:23:06Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| ac8ae8a9-2a22-4735-861b-0660c60b044f | overcloud-ControllerCephDeployment-adxqq3scn3wz                                                               | CREATE_COMPLETE    | 2015-08-21T06:23:12Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4f0449ff-c3fe-4af1-907a-15fd35f4447e | overcloud-ControllerAllNodesDeployment-kpfs37qjm7n6                                                           | CREATE_COMPLETE    | 2015-08-21T06:23:17Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 734ae360-5720-4248-96de-fcc53ab24a12 | overcloud-ComputeCephDeployment-mlqwqifsrfrb                                                                  | CREATE_COMPLETE    | 2015-08-21T06:23:19Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 271aa9ef-a118-48eb-b4d1-92c20538d38e | overcloud-CephStorageCephDeployment-6zw6ajmb3qpp                                                              | CREATE_COMPLETE    | 2015-08-21T06:23:23Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 0a5a1ce9-7a59-4dd3-a501-4e4859d0fb23 | overcloud-ObjectStorageAllNodesDeployment-htmvldloqm3f                                                        | CREATE_COMPLETE    | 2015-08-21T06:23:25Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 6a559898-0cdc-4337-835d-601b41b9cb1c | overcloud-ComputeAllNodesDeployment-mgrucunikgpb                                                              | CREATE_COMPLETE    | 2015-08-21T06:23:28Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f5ac100a-05c1-4994-973b-4fdbbe45ec9d | overcloud-BlockStorageAllNodesDeployment-pvgrw2c2dnf6                                                         | CREATE_COMPLETE    | 2015-08-21T06:23:31Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 10d908a7-82ec-41c5-b7b7-40278b72f759 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e                                                       | CREATE_COMPLETE    | 2015-08-21T06:23:37Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 55c0b272-4c85-4318-a816-341f72f36899 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-StorageDeployment_Step1-ayikgu7mbjjy                  | CREATE_COMPLETE    | 2015-08-21T06:23:40Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| 6568fbe9-22fd-4d3a-b273-351bd1c0f180 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-StorageRingbuilderDeployment_Step2-g2bfxw743vyl       | CREATE_COMPLETE    | 2015-08-21T06:23:44Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| ccf75bf8-825a-4d79-9a3a-3abf403dfe84 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-ExtraConfig-grkpa5cyaj4j                              | CREATE_COMPLETE    | 2015-08-21T06:23:49Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| 10b8ea8d-89b7-47b8-8db4-59783222d2c8 | overcloud-ComputeNodesPostDeployment-sj3pzjzcnexa                                                             | CREATE_IN_PROGRESS | 2015-08-21T06:24:34Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f5125554-08c1-4e3f-a5c0-4efcb20a03fb | overcloud-ComputeNodesPostDeployment-sj3pzjzcnexa-ComputePuppetDeployment-jmdrg76fgnkk                        | CREATE_IN_PROGRESS | 2015-08-21T06:24:37Z | 10b8ea8d-89b7-47b8-8db4-59783222d2c8 |
| 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs                                                          | CREATE_IN_PROGRESS | 2015-08-21T06:25:17Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| caf1fca4-953d-4cde-b257-83be74837557 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerPuppetConfig-5a6vhoyawz7n                      | CREATE_COMPLETE    | 2015-08-21T06:25:19Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 8aefd1d4-3b0f-4db2-bd82-b45d256a22f2 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerLoadBalancerDeployment_Step1-wx347abkxfah      | CREATE_COMPLETE    | 2015-08-21T06:25:24Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 0796df35-716c-40e0-b52b-35cb157bca82 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerServicesBaseDeployment_Step2-uwysertxhbs5      | CREATE_COMPLETE    | 2015-08-21T06:28:17Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| e28483ee-58fc-4fcf-b83b-32fb82273849 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerRingbuilderDeployment_Step3-crgoagoipcqu       | CREATE_COMPLETE    | 2015-08-21T06:31:31Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 5b35e391-34dc-4c51-9ba0-68cab3be97c4 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerOvercloudServicesDeployment_Step4-7dbmd7npeax4 | CREATE_COMPLETE    | 2015-08-21T06:33:03Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 5c97b3ed-7ef0-4b4a-a977-be30914fe4f5 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerOvercloudServicesDeployment_Step5-juj37e7skuvx | CREATE_IN_PROGRESS | 2015-08-21T06:40:47Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+

Comment 40 Mike Burns 2015-08-21 13:32:54 UTC

This bug is closed.  I've cloned it to a separate bug (Bug 1255759).  Please make any further updates there and not in this bug.

Note You need to log in before you can comment on or make changes to this bug.

augol
fhubik
gfidente
ggillies
jliberma
jschluet
jstransk
lnatapov
mburns
mcornea
nbarcet
ohochman
opavlenk
rhel-osp-director-maint
rhos-maint
rlandy
sasha
sbaker
shardy
yeylon