Bug 1255759

Summary: heat race condition causing deployments to get stuck at various places
Product: Red Hat OpenStack Reporter: Mike Burns <mburns>
Component: openstack-heatAssignee: Steve Baker <sbaker>
Status: CLOSED ERRATA QA Contact: Amit Ugol <augol>
Severity: high Docs Contact:
Priority: high    
Version: DirectorCC: augol, ddomingo, djuran, fhubik, gfidente, ggillies, jliberma, jschluet, jstransk, kprabhak, lnatapov, mburns, mcornea, nbarcet, ohochman, opavlenk, rhel-osp-director-maint, rhos-maint, rlandy, sasha, sbaker, shardy, yeylon, zbitter
Target Milestone: z2Keywords: Triaged, ZStream
Target Release: 7.0 (Kilo)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-heat-2015.1.1-1.el7ost Doc Type: Bug Fix
Doc Text:
Previously, it was possible for deployment signalling to cause other deployments to remain in an IN_PROGRESS state until stack timeout. This prevented deployment data from reacing overcloud nodes, thereby causing the overcloud deployment to time out and go into a FAILED state. This issue was caused by legacy code paths, which update the metadata on every stack resource after any signal. These paths have now been disabled; as such, metadata is only updated using the resource-level locking mechanism. This mechanism correctly handles concurrent metadata updates.
Story Points: ---
Clone Of: 1249628 Environment:
Last Closed: 2015-10-08 12:20:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1258497    

Description Mike Burns 2015-08-21 13:31:37 UTC
+++ This bug was initially created as a clone of Bug #1249628 +++


--- Additional comment from Steve Baker on 2015-08-20 18:09:59 EDT ---

The previous fixes should have fixed *a* race but it looks like there may be another issue which is still causing this bug. I have a logging patch which would help with diagnosing where the issue is.

If you have an environment which is affected, please do the following
- apply service_software_config_logging.patch to /usr/lib/python2.7/site-packages/heat/engine/service_software_config.py
- in /etc/heat/heat.conf set the following
  [DEFAULT] debug = False
  [database] connection_debug = 50
- truncate --size 0 /var/log/heat/heat-engine.log
- systemctl restart openstack-heat-engine
- deploy overcloud until problem recurs
- Make /var/log/heat/heat-engine.log available as a comment in this bug

--- Additional comment from Graeme Gillies on 2015-08-21 02:49:57 EDT ---

Hi,

Reproduced problem with the patch from Comment 37 though unfortunately I had sql debugging turned off.

It failed at Step5

Attached is log

I'll try to reproduce with sql logging turned on as well and attach

Regards,

Graeme

--- Additional comment from Graeme Gillies on 2015-08-21 02:50:54 EDT ---

output of my stack list as well to help match to log

$ heat stack-list --show-nested
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+
| id                                   | stack_name                                                                                                    | stack_status       | creation_time        | parent                               |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+
| 1d0744f2-a3db-4f1f-b68e-7281352e7a17 | overcloud                                                                                                     | CREATE_IN_PROGRESS | 2015-08-21T05:57:47Z | None                                 |
| 37b9045a-67f9-4def-b97a-1a8677f91fbd | overcloud-Networks-iuunei3g4d7i                                                                               | CREATE_COMPLETE    | 2015-08-21T05:57:50Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 748c743f-6f0a-4bc0-9306-6c9a64b01060 | overcloud-Networks-iuunei3g4d7i-StorageNetwork-qwz2kqao2xqe                                                   | CREATE_COMPLETE    | 2015-08-21T05:57:51Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| da23de84-21ae-4e5d-930a-3a9ea9589885 | overcloud-VipConfig-56czet7rcdlv                                                                              | CREATE_COMPLETE    | 2015-08-21T05:57:51Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| c749e566-dd81-4830-bf2b-a399d5400ac6 | overcloud-Networks-iuunei3g4d7i-StorageMgmtNetwork-forc3jbtk7vw                                               | CREATE_COMPLETE    | 2015-08-21T05:57:52Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| f70b8abd-e6a3-4693-8928-5043377cac69 | overcloud-Networks-iuunei3g4d7i-InternalNetwork-lo4zyz2i5uiw                                                  | CREATE_COMPLETE    | 2015-08-21T05:57:53Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| e947cff1-6b66-443c-a809-0d5432e83f1e | overcloud-Networks-iuunei3g4d7i-ExternalNetwork-qpnukszxw5tc                                                  | CREATE_COMPLETE    | 2015-08-21T05:57:54Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| e22fddb9-be88-49f3-aa1e-7edb4dd5d8aa | overcloud-Networks-iuunei3g4d7i-TenantNetwork-a5wucf74h5bg                                                    | CREATE_COMPLETE    | 2015-08-21T05:57:56Z | 37b9045a-67f9-4def-b97a-1a8677f91fbd |
| 4a125772-6032-4f04-b163-0b9a9020620e | overcloud-ObjectStorage-v5nm44zukdrb                                                                          | CREATE_COMPLETE    | 2015-08-21T05:58:02Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 987f51eb-ed3f-45ec-af6c-febf047488a3 | overcloud-CephStorage-72trllu54fed                                                                            | CREATE_COMPLETE    | 2015-08-21T05:58:04Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4065b9ea-1182-46d6-b8f5-0e84e8189859 | overcloud-RedisVirtualIP-fldb36xjm7b2                                                                         | CREATE_COMPLETE    | 2015-08-21T05:58:08Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| da53a9c0-27f8-40cd-997b-dc827abaeef1 | overcloud-StorageMgmtVirtualIP-ossak4ifhwv7                                                                   | CREATE_COMPLETE    | 2015-08-21T05:58:09Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 16ac36a4-b922-4ede-89bd-32dcccfa7f4f | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh                                                             | CREATE_COMPLETE    | 2015-08-21T05:58:10Z | 987f51eb-ed3f-45ec-af6c-febf047488a3 |
| 8349805b-c043-4b1b-b388-40c46a527e2d | overcloud-PublicVirtualIP-3nbhyn4viqol                                                                        | CREATE_COMPLETE    | 2015-08-21T05:58:11Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f301cd61-a8d8-41cc-ba84-4281541e2b15 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-UpdateConfig-shtbycdgyses                                   | CREATE_COMPLETE    | 2015-08-21T05:58:11Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 6d54e85f-35d2-4f54-9dbb-769f04ada11b | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NodeUserData-ien5lw6zjykw                                   | CREATE_COMPLETE    | 2015-08-21T05:58:12Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| cd9bb970-b0c8-49d9-a564-7de3fb56a517 | overcloud-StorageVirtualIP-pumvnrdn2etp                                                                       | CREATE_COMPLETE    | 2015-08-21T05:58:13Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 252a8809-ce00-4a9d-8088-891447efad55 | overcloud-InternalApiVirtualIP-c4c5qezsqfwn                                                                   | CREATE_COMPLETE    | 2015-08-21T05:58:15Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f81bcdf3-21d6-41f0-84eb-6d529af4b6da | overcloud-VipMap-van73sfst7kt                                                                                 | CREATE_COMPLETE    | 2015-08-21T05:58:20Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 3be18f3d-28a4-4b7f-aa49-3d1f55f61dcc | overcloud-Compute-gp2hrbniskdh                                                                                | CREATE_COMPLETE    | 2015-08-21T05:58:25Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 22d8dd30-012f-4fe9-8c53-01a69193bd4b | overcloud-Controller-5rcadg5ssquk                                                                             | CREATE_COMPLETE    | 2015-08-21T05:58:31Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah                                                                 | CREATE_COMPLETE    | 2015-08-21T05:58:34Z | 3be18f3d-28a4-4b7f-aa49-3d1f55f61dcc |
| 1cb132da-ee9f-4488-8865-6864657e557f | overcloud-BlockStorage-wgi7df3xot25                                                                           | CREATE_COMPLETE    | 2015-08-21T05:58:36Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 5666450e-3a1e-4e95-9baf-414b27640622 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-UpdateConfig-zgy3wxxkxzwj                                       | CREATE_COMPLETE    | 2015-08-21T05:58:38Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 7b744a27-144e-4ab1-990a-2ab18cd8f669 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NodeUserData-2srxjq24dzli                                       | CREATE_COMPLETE    | 2015-08-21T05:58:40Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 4a93ae08-58c5-4704-bf56-a260e483b61e | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u                                                              | CREATE_COMPLETE    | 2015-08-21T05:58:49Z | 22d8dd30-012f-4fe9-8c53-01a69193bd4b |
| 72dc0ac8-53ba-4778-bfe4-b758aae9b933 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NodeUserData-dojuhv37dei6                                    | CREATE_COMPLETE    | 2015-08-21T05:58:56Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 23eb057e-6597-4857-85a4-9d9db9e280b1 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-UpdateConfig-d3rvqgbwol3o                                    | CREATE_COMPLETE    | 2015-08-21T05:59:01Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 65b9ad48-a2b6-4e20-b8f2-404390a03b82 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-StorageMgmtPort-uugaoc3sppyj                                | CREATE_COMPLETE    | 2015-08-21T06:17:05Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 3a103c70-8719-4a03-b5a2-e50236d08eec | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-StoragePort-dizfwzvwskyk                                    | CREATE_COMPLETE    | 2015-08-21T06:17:15Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 95bf01c8-f4b9-4738-b21a-c98dc54c592f | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetworkConfig-y4ghztkohfh7                                  | CREATE_COMPLETE    | 2015-08-21T06:17:25Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| f4d35c64-3916-4ff3-bdb4-ed347f58ccf0 | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetIpMap-ium4vgx4sd53                                       | CREATE_COMPLETE    | 2015-08-21T06:17:33Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 2564b862-a1be-4a2a-92ae-96cd9be9fc0d | overcloud-CephStorage-72trllu54fed-0-hltnrjgu72sh-NetIpSubnetMap-wi5pko7qsa6l                                 | CREATE_COMPLETE    | 2015-08-21T06:17:35Z | 16ac36a4-b922-4ede-89bd-32dcccfa7f4f |
| 83d16435-f9ee-46c8-8569-a914ab2966c9 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-TenantPort-v7j5vfl5wrd2                                         | CREATE_COMPLETE    | 2015-08-21T06:17:58Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| a1cfe06c-bac4-4dc5-9663-012661451c8b | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-StoragePort-lnvmgslwglng                                        | CREATE_COMPLETE    | 2015-08-21T06:18:01Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| fd832499-736d-4905-97ec-85efcbd367bd | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-InternalApiPort-23jfu3zmefwv                                    | CREATE_COMPLETE    | 2015-08-21T06:18:05Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 5df6f8a6-b104-4eca-b155-5982ea15acd7 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NetIpMap-v6ny5zct6f7s                                           | CREATE_COMPLETE    | 2015-08-21T06:18:11Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| e1d62c12-4f03-421b-b22f-8fa127be8060 | overcloud-Compute-gp2hrbniskdh-0-464uypjamcah-NetworkConfig-msw4lwut5z7g                                      | CREATE_COMPLETE    | 2015-08-21T06:18:13Z | fdb600ed-b3fa-4f0f-ae7a-bb7aeb684b13 |
| 339d5aad-70f7-42c3-afa3-deef4f6def71 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-TenantPort-4bw5djiof7n5                                      | CREATE_COMPLETE    | 2015-08-21T06:18:14Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| a2dab192-11ba-48f9-873b-bc45a6304e82 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-ExternalPort-fjoqr42svs3h                                    | CREATE_COMPLETE    | 2015-08-21T06:18:18Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 7017e935-fc85-42a1-9b40-520b15a434a7 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-StorageMgmtPort-xn4jeyizhas3                                 | CREATE_COMPLETE    | 2015-08-21T06:18:22Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 708eb94d-83f1-41d4-82f7-a95132c6ff96 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-InternalApiPort-nfw465dpjkiw                                 | CREATE_COMPLETE    | 2015-08-21T06:18:27Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 2e05687d-6200-47be-8b3f-95b52b524a45 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-StoragePort-fn3ohp4oflka                                     | CREATE_COMPLETE    | 2015-08-21T06:18:30Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| b52572cc-5c57-4c6f-b182-25f858557966 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetIpSubnetMap-6a6cjlotsect                                  | CREATE_COMPLETE    | 2015-08-21T06:18:41Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 561a6ec7-9b47-41bb-a291-518a4235500e | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetworkConfig-o6hvbchm4egu                                   | CREATE_COMPLETE    | 2015-08-21T06:18:44Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 62500da2-e04a-4b12-9019-12a0e4c3c9ec | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-NetIpMap-mhcjq6slbief                                        | CREATE_COMPLETE    | 2015-08-21T06:18:47Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| 48b6805d-5c94-4d1c-8d63-e48f17970898 | overcloud-Controller-5rcadg5ssquk-0-qmqy3kkokj4u-ControllerExtraConfigPre-bwysoleuh7bg                        | CREATE_COMPLETE    | 2015-08-21T06:22:31Z | 4a93ae08-58c5-4704-bf56-a260e483b61e |
| d7972546-81ec-4862-a456-abfec868297e | overcloud-ControllerBootstrapNodeConfig-ll3enwdxdhcr                                                          | CREATE_COMPLETE    | 2015-08-21T06:22:36Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 68e6ae96-37ac-445b-bdb4-fe0fa023fd9c | overcloud-ControllerIpListMap-ta4r45cszs2r                                                                    | CREATE_COMPLETE    | 2015-08-21T06:22:39Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| ac918a51-f06a-4e5c-94e2-2b252db30008 | overcloud-VipDeployment-dnbfv45aypxc                                                                          | CREATE_COMPLETE    | 2015-08-21T06:22:42Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| d5cf377f-0acc-420e-b8ae-fc8d82c10003 | overcloud-SwiftDevicesAndProxyConfig-2oid4zqloy3j                                                             | CREATE_COMPLETE    | 2015-08-21T06:22:44Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 462a4a30-184f-49b1-b15b-77525f6b5017 | overcloud-ControllerBootstrapNodeDeployment-5j56m5ayrzem                                                      | CREATE_COMPLETE    | 2015-08-21T06:22:48Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| bc6cb3d0-849e-4c38-856b-26113bf8320e | overcloud-ControllerClusterDeployment-g4wr453gjmg3                                                            | CREATE_COMPLETE    | 2015-08-21T06:22:50Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 03b6eb84-4115-4ab1-9a38-15ce432068b3 | overcloud-ControllerSwiftDeployment-zhzes7y6kgmn                                                              | CREATE_COMPLETE    | 2015-08-21T06:22:53Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| fe6fe74e-cba8-48bb-a8f6-7703b326d627 | overcloud-ObjectStorageSwiftDeployment-aumsowahccrx                                                           | CREATE_COMPLETE    | 2015-08-21T06:22:55Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4e16b903-1664-4695-b54d-db690bb9e76b | overcloud-allNodesConfig-3izzrpt4oava                                                                         | CREATE_COMPLETE    | 2015-08-21T06:22:58Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 653149ab-11f3-4481-bad7-cf369d04795c | overcloud-CephClusterConfig-dj3p7hnjotcl                                                                      | CREATE_COMPLETE    | 2015-08-21T06:22:59Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 8c2b6239-5b1d-41a6-b0ae-df36bac5d5f0 | overcloud-CephStorageAllNodesDeployment-4goz26gg6vut                                                          | CREATE_COMPLETE    | 2015-08-21T06:23:06Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| ac8ae8a9-2a22-4735-861b-0660c60b044f | overcloud-ControllerCephDeployment-adxqq3scn3wz                                                               | CREATE_COMPLETE    | 2015-08-21T06:23:12Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 4f0449ff-c3fe-4af1-907a-15fd35f4447e | overcloud-ControllerAllNodesDeployment-kpfs37qjm7n6                                                           | CREATE_COMPLETE    | 2015-08-21T06:23:17Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 734ae360-5720-4248-96de-fcc53ab24a12 | overcloud-ComputeCephDeployment-mlqwqifsrfrb                                                                  | CREATE_COMPLETE    | 2015-08-21T06:23:19Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 271aa9ef-a118-48eb-b4d1-92c20538d38e | overcloud-CephStorageCephDeployment-6zw6ajmb3qpp                                                              | CREATE_COMPLETE    | 2015-08-21T06:23:23Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 0a5a1ce9-7a59-4dd3-a501-4e4859d0fb23 | overcloud-ObjectStorageAllNodesDeployment-htmvldloqm3f                                                        | CREATE_COMPLETE    | 2015-08-21T06:23:25Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 6a559898-0cdc-4337-835d-601b41b9cb1c | overcloud-ComputeAllNodesDeployment-mgrucunikgpb                                                              | CREATE_COMPLETE    | 2015-08-21T06:23:28Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f5ac100a-05c1-4994-973b-4fdbbe45ec9d | overcloud-BlockStorageAllNodesDeployment-pvgrw2c2dnf6                                                         | CREATE_COMPLETE    | 2015-08-21T06:23:31Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 10d908a7-82ec-41c5-b7b7-40278b72f759 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e                                                       | CREATE_COMPLETE    | 2015-08-21T06:23:37Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| 55c0b272-4c85-4318-a816-341f72f36899 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-StorageDeployment_Step1-ayikgu7mbjjy                  | CREATE_COMPLETE    | 2015-08-21T06:23:40Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| 6568fbe9-22fd-4d3a-b273-351bd1c0f180 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-StorageRingbuilderDeployment_Step2-g2bfxw743vyl       | CREATE_COMPLETE    | 2015-08-21T06:23:44Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| ccf75bf8-825a-4d79-9a3a-3abf403dfe84 | overcloud-ObjectStorageNodesPostDeployment-covazqti427e-ExtraConfig-grkpa5cyaj4j                              | CREATE_COMPLETE    | 2015-08-21T06:23:49Z | 10d908a7-82ec-41c5-b7b7-40278b72f759 |
| 10b8ea8d-89b7-47b8-8db4-59783222d2c8 | overcloud-ComputeNodesPostDeployment-sj3pzjzcnexa                                                             | CREATE_IN_PROGRESS | 2015-08-21T06:24:34Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| f5125554-08c1-4e3f-a5c0-4efcb20a03fb | overcloud-ComputeNodesPostDeployment-sj3pzjzcnexa-ComputePuppetDeployment-jmdrg76fgnkk                        | CREATE_IN_PROGRESS | 2015-08-21T06:24:37Z | 10b8ea8d-89b7-47b8-8db4-59783222d2c8 |
| 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs                                                          | CREATE_IN_PROGRESS | 2015-08-21T06:25:17Z | 1d0744f2-a3db-4f1f-b68e-7281352e7a17 |
| caf1fca4-953d-4cde-b257-83be74837557 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerPuppetConfig-5a6vhoyawz7n                      | CREATE_COMPLETE    | 2015-08-21T06:25:19Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 8aefd1d4-3b0f-4db2-bd82-b45d256a22f2 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerLoadBalancerDeployment_Step1-wx347abkxfah      | CREATE_COMPLETE    | 2015-08-21T06:25:24Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 0796df35-716c-40e0-b52b-35cb157bca82 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerServicesBaseDeployment_Step2-uwysertxhbs5      | CREATE_COMPLETE    | 2015-08-21T06:28:17Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| e28483ee-58fc-4fcf-b83b-32fb82273849 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerRingbuilderDeployment_Step3-crgoagoipcqu       | CREATE_COMPLETE    | 2015-08-21T06:31:31Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 5b35e391-34dc-4c51-9ba0-68cab3be97c4 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerOvercloudServicesDeployment_Step4-7dbmd7npeax4 | CREATE_COMPLETE    | 2015-08-21T06:33:03Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
| 5c97b3ed-7ef0-4b4a-a977-be30914fe4f5 | overcloud-ControllerNodesPostDeployment-awf2wmyeljhs-ControllerOvercloudServicesDeployment_Step5-juj37e7skuvx | CREATE_IN_PROGRESS | 2015-08-21T06:40:47Z | 8bb7505e-2997-4a3c-8a2a-a81e0fd60ad8 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------------------------------+

Comment 7 Graeme Gillies 2015-08-28 03:31:33 UTC
Been testing the proposed patch by sbaker for multiple days now and can't reproduce the issue at all, so I consider this patch having completely fixed the issue

Regards,

Graeme

Comment 11 Amit Ugol 2015-09-13 07:16:38 UTC
I have already closed on of these too soon. I would like to keep this on qa for a while longer (not blocking ATM). I will gather results from CI, virt setups and HA BM setups and will verify it when I have more data. This might take up to a week.

Comment 12 Amit Ugol 2015-09-22 08:39:31 UTC
So far so good, I am clicking with a shaky finger on the Save Changes...

Comment 14 errata-xmlrpc 2015-10-08 12:20:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1865