Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1261129

Summary:	How to recover heat stack-list when installation failed
Product:	Red Hat OpenStack	Reporter:	bigswitch <rhosp-bugs-internal>
Component:	rhosp-director	Assignee:	Zane Bitter <zbitter>
Status:	CLOSED DUPLICATE	QA Contact:	yeylon <yeylon>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	mburns, rhel-osp-director-maint, rhosp-bugs-internal, srevivo
Target Milestone:	---
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-09-22 17:19:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description bigswitch 2015-09-08 16:21:49 UTC

Description of problem:
When attempting to scale out compute nodes deployment, it will fail and heat stack-list will show update_failed. Under this situation, how do I recover heat such that I can continue to attempt to deploy more compute nodes. heat stack-delete overcloud will delete everything including the previously deployed controller and compute nodes.
if I retry the deployment it will stay as failed and I am not sure if I will be able to get a successful deployment later.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Deploy 3 controller and 1 compute
2. Progresively add more compute nodes, it will fail at some point


Actual results:


Expected results:
There should be a way to rollback to the last known good state?

Additional info:

Comment 3 bigswitch 2015-09-10 16:43:35 UTC

When the deployment failed, and I deleted of the failed (error) nodes from the failed deployment using this command:


openstack overcloud node delete -e /home/stack/templates/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --templates --stack 34651110-aea9-4665-9070-c095a1b1fe9e b206185f-3caf-4c89-ac07-aba04f541d76  <<<<< stackid and error node id

[stack@dell-undercloud ~]$   heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_IN_PROGRESS | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@dell-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 08c5e611-4885-412f-8641-a4452a8bdcd5 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| 1ff336ae-8d73-4893-b970-d2dce66176b0 | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | overcloud-compute-10   | ACTIVE | -          | Running     | ctlplane=192.0.2.49 |
| bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | overcloud-compute-11   | ACTIVE | -          | Running     | ctlplane=192.0.2.53 |
| f0c24cbc-d018-42d4-a18e-fc320f8fdaba | overcloud-compute-12   | ACTIVE | -          | Running     | ctlplane=192.0.2.47 |
| 946a09a4-b094-405a-9794-63755ea49648 | overcloud-compute-13   | ACTIVE | -          | Running     | ctlplane=192.0.2.46 |
| 058f4064-b149-4160-bf94-813f1d755480 | overcloud-compute-15   | ACTIVE | -          | Running     | ctlplane=192.0.2.43 |
| 4ec8f5d0-9773-469c-b5f5-35c8cc732040 | overcloud-compute-16   | ACTIVE | -          | Running     | ctlplane=192.0.2.52 |
| 4fd49237-b58c-4532-98e7-939dff3ea8a6 | overcloud-compute-17   | ACTIVE | -          | Running     | ctlplane=192.0.2.45 |
| b6ed3c04-9064-4555-90ea-c1b06b27da00 | overcloud-compute-18   | ACTIVE | -          | Running     | ctlplane=192.0.2.48 |
| 1037d111-e4e1-45b0-9a29-e25c63f43d1d | overcloud-compute-19   | ACTIVE | -          | Running     | ctlplane=192.0.2.51 |
| 458c6e45-c0dc-44ed-8be8-f082034e2427 | overcloud-compute-2    | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 61bc3762-4bb2-43e2-8116-94267dc21280 | overcloud-compute-20   | ACTIVE | -          | Running     | ctlplane=192.0.2.50 |
| cd9f42dc-3e8e-48f3-841b-335338407c5b | overcloud-compute-21   | ERROR  | -          | NOSTATE     |                     |
| a0af2f73-86e2-42b0-be81-01488afe512e | overcloud-compute-22   | ACTIVE | -          | Running     | ctlplane=192.0.2.56 |
| 4a42093e-86f4-43a4-9c07-a0402f9f94c3 | overcloud-compute-23   | ACTIVE | -          | Running     | ctlplane=192.0.2.54 |
| b206185f-3caf-4c89-ac07-aba04f541d76 | overcloud-compute-24   | ERROR  | -          | NOSTATE     |                     |
| d2a225c1-5a63-4d68-b38c-a083f5712d6e | overcloud-compute-25   | ACTIVE | -          | Running     | ctlplane=192.0.2.59 |
| 19505757-e28e-4aa1-88b2-019c5b7839b4 | overcloud-compute-26   | ERROR  | -          | NOSTATE     |                     |
| 0fcf2976-0505-4cc9-b1e4-2a3d8d1569a2 | overcloud-compute-27   | ACTIVE | -          | Running     | ctlplane=192.0.2.57 |
| cac3a6f2-8b70-4834-9ff9-9e7fe44e2dea | overcloud-compute-28   | ACTIVE | -          | Running     | ctlplane=192.0.2.58 |
| 4b1faad4-2e38-4ffc-a3c8-6b536adc9159 | overcloud-compute-29   | ERROR  | -          | NOSTATE     |                     |
| 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | overcloud-compute-3    | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| 46fedd21-f54b-4e8d-9d94-207cec96d37b | overcloud-compute-30   | ACTIVE | -          | Running     | ctlplane=192.0.2.55 |
| a19bb1d0-8b0c-4276-9323-68b2b07938a2 | overcloud-compute-4    | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | overcloud-compute-5    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| 21851e51-c329-43ad-bdc2-403ce9c38eb2 | overcloud-compute-6    | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
| e1f24a55-0481-4a9d-844d-24bb8f792cae | overcloud-compute-7    | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| c87353f1-d60e-4cee-9e0a-706902030992 | overcloud-compute-8    | ACTIVE | -          | Running     | ctlplane=192.0.2.25 |
| 040f55f4-28a4-448f-b2ff-864d26812bb1 | overcloud-compute-9    | ACTIVE | -          | Running     | ctlplane=192.0.2.44 |
| e02db87b-8b83-4f3f-abef-67597eaed760 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 72d56c64-96d0-432f-bdd6-012d76c27659 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| d6efae4d-d76c-441f-93e8-48411f6e23c2 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+

[stack@dell-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 08c5e611-4885-412f-8641-a4452a8bdcd5 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| 1ff336ae-8d73-4893-b970-d2dce66176b0 | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | overcloud-compute-10   | ACTIVE | -          | Running     | ctlplane=192.0.2.49 |
| bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | overcloud-compute-11   | ACTIVE | -          | Running     | ctlplane=192.0.2.53 |
| f0c24cbc-d018-42d4-a18e-fc320f8fdaba | overcloud-compute-12   | ACTIVE | -          | Running     | ctlplane=192.0.2.47 |
| 946a09a4-b094-405a-9794-63755ea49648 | overcloud-compute-13   | ACTIVE | -          | Running     | ctlplane=192.0.2.46 |
| 058f4064-b149-4160-bf94-813f1d755480 | overcloud-compute-15   | ACTIVE | -          | Running     | ctlplane=192.0.2.43 |
| 4ec8f5d0-9773-469c-b5f5-35c8cc732040 | overcloud-compute-16   | ACTIVE | -          | Running     | ctlplane=192.0.2.52 |
| 4fd49237-b58c-4532-98e7-939dff3ea8a6 | overcloud-compute-17   | ACTIVE | -          | Running     | ctlplane=192.0.2.45 |
| b6ed3c04-9064-4555-90ea-c1b06b27da00 | overcloud-compute-18   | ACTIVE | -          | Running     | ctlplane=192.0.2.48 |
| 1037d111-e4e1-45b0-9a29-e25c63f43d1d | overcloud-compute-19   | ACTIVE | -          | Running     | ctlplane=192.0.2.51 |
| 458c6e45-c0dc-44ed-8be8-f082034e2427 | overcloud-compute-2    | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | overcloud-compute-3    | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| a19bb1d0-8b0c-4276-9323-68b2b07938a2 | overcloud-compute-4    | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | overcloud-compute-5    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| 21851e51-c329-43ad-bdc2-403ce9c38eb2 | overcloud-compute-6    | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
| e1f24a55-0481-4a9d-844d-24bb8f792cae | overcloud-compute-7    | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| c87353f1-d60e-4cee-9e0a-706902030992 | overcloud-compute-8    | ACTIVE | -          | Running     | ctlplane=192.0.2.25 |
| 040f55f4-28a4-448f-b2ff-864d26812bb1 | overcloud-compute-9    | ACTIVE | -          | Running     | ctlplane=192.0.2.44 |
| e02db87b-8b83-4f3f-abef-67597eaed760 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 72d56c64-96d0-432f-bdd6-012d76c27659 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| d6efae4d-d76c-441f-93e8-48411f6e23c2 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_IN_PROGRESS | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_IN_PROGRESS | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_COMPLETE | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+-----------------+----------------------+
[stack@dell-undercloud ~]$


It proceed to delete all compute nodes deployed during the last failed exercise including those whose state is active, and return heat stack to update_complete state
Is this the workflow to recover heat stack?

Comment 4 bigswitch 2015-09-10 16:43:54 UTC

When the deployment failed, and I deleted of the failed (error) nodes from the failed deployment using this command:


openstack overcloud node delete -e /home/stack/templates/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --templates --stack 34651110-aea9-4665-9070-c095a1b1fe9e b206185f-3caf-4c89-ac07-aba04f541d76  <<<<< stackid and error node id

[stack@dell-undercloud ~]$   heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_IN_PROGRESS | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@dell-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 08c5e611-4885-412f-8641-a4452a8bdcd5 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| 1ff336ae-8d73-4893-b970-d2dce66176b0 | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | overcloud-compute-10   | ACTIVE | -          | Running     | ctlplane=192.0.2.49 |
| bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | overcloud-compute-11   | ACTIVE | -          | Running     | ctlplane=192.0.2.53 |
| f0c24cbc-d018-42d4-a18e-fc320f8fdaba | overcloud-compute-12   | ACTIVE | -          | Running     | ctlplane=192.0.2.47 |
| 946a09a4-b094-405a-9794-63755ea49648 | overcloud-compute-13   | ACTIVE | -          | Running     | ctlplane=192.0.2.46 |
| 058f4064-b149-4160-bf94-813f1d755480 | overcloud-compute-15   | ACTIVE | -          | Running     | ctlplane=192.0.2.43 |
| 4ec8f5d0-9773-469c-b5f5-35c8cc732040 | overcloud-compute-16   | ACTIVE | -          | Running     | ctlplane=192.0.2.52 |
| 4fd49237-b58c-4532-98e7-939dff3ea8a6 | overcloud-compute-17   | ACTIVE | -          | Running     | ctlplane=192.0.2.45 |
| b6ed3c04-9064-4555-90ea-c1b06b27da00 | overcloud-compute-18   | ACTIVE | -          | Running     | ctlplane=192.0.2.48 |
| 1037d111-e4e1-45b0-9a29-e25c63f43d1d | overcloud-compute-19   | ACTIVE | -          | Running     | ctlplane=192.0.2.51 |
| 458c6e45-c0dc-44ed-8be8-f082034e2427 | overcloud-compute-2    | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 61bc3762-4bb2-43e2-8116-94267dc21280 | overcloud-compute-20   | ACTIVE | -          | Running     | ctlplane=192.0.2.50 |
| cd9f42dc-3e8e-48f3-841b-335338407c5b | overcloud-compute-21   | ERROR  | -          | NOSTATE     |                     |
| a0af2f73-86e2-42b0-be81-01488afe512e | overcloud-compute-22   | ACTIVE | -          | Running     | ctlplane=192.0.2.56 |
| 4a42093e-86f4-43a4-9c07-a0402f9f94c3 | overcloud-compute-23   | ACTIVE | -          | Running     | ctlplane=192.0.2.54 |
| b206185f-3caf-4c89-ac07-aba04f541d76 | overcloud-compute-24   | ERROR  | -          | NOSTATE     |                     |
| d2a225c1-5a63-4d68-b38c-a083f5712d6e | overcloud-compute-25   | ACTIVE | -          | Running     | ctlplane=192.0.2.59 |
| 19505757-e28e-4aa1-88b2-019c5b7839b4 | overcloud-compute-26   | ERROR  | -          | NOSTATE     |                     |
| 0fcf2976-0505-4cc9-b1e4-2a3d8d1569a2 | overcloud-compute-27   | ACTIVE | -          | Running     | ctlplane=192.0.2.57 |
| cac3a6f2-8b70-4834-9ff9-9e7fe44e2dea | overcloud-compute-28   | ACTIVE | -          | Running     | ctlplane=192.0.2.58 |
| 4b1faad4-2e38-4ffc-a3c8-6b536adc9159 | overcloud-compute-29   | ERROR  | -          | NOSTATE     |                     |
| 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | overcloud-compute-3    | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| 46fedd21-f54b-4e8d-9d94-207cec96d37b | overcloud-compute-30   | ACTIVE | -          | Running     | ctlplane=192.0.2.55 |
| a19bb1d0-8b0c-4276-9323-68b2b07938a2 | overcloud-compute-4    | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | overcloud-compute-5    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| 21851e51-c329-43ad-bdc2-403ce9c38eb2 | overcloud-compute-6    | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
| e1f24a55-0481-4a9d-844d-24bb8f792cae | overcloud-compute-7    | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| c87353f1-d60e-4cee-9e0a-706902030992 | overcloud-compute-8    | ACTIVE | -          | Running     | ctlplane=192.0.2.25 |
| 040f55f4-28a4-448f-b2ff-864d26812bb1 | overcloud-compute-9    | ACTIVE | -          | Running     | ctlplane=192.0.2.44 |
| e02db87b-8b83-4f3f-abef-67597eaed760 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 72d56c64-96d0-432f-bdd6-012d76c27659 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| d6efae4d-d76c-441f-93e8-48411f6e23c2 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+

[stack@dell-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 08c5e611-4885-412f-8641-a4452a8bdcd5 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| 1ff336ae-8d73-4893-b970-d2dce66176b0 | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| db1f43c2-95ba-4d9e-a6d3-2bbd1cc26923 | overcloud-compute-10   | ACTIVE | -          | Running     | ctlplane=192.0.2.49 |
| bdcaa93c-ed55-496b-a38b-b7bbb05b4677 | overcloud-compute-11   | ACTIVE | -          | Running     | ctlplane=192.0.2.53 |
| f0c24cbc-d018-42d4-a18e-fc320f8fdaba | overcloud-compute-12   | ACTIVE | -          | Running     | ctlplane=192.0.2.47 |
| 946a09a4-b094-405a-9794-63755ea49648 | overcloud-compute-13   | ACTIVE | -          | Running     | ctlplane=192.0.2.46 |
| 058f4064-b149-4160-bf94-813f1d755480 | overcloud-compute-15   | ACTIVE | -          | Running     | ctlplane=192.0.2.43 |
| 4ec8f5d0-9773-469c-b5f5-35c8cc732040 | overcloud-compute-16   | ACTIVE | -          | Running     | ctlplane=192.0.2.52 |
| 4fd49237-b58c-4532-98e7-939dff3ea8a6 | overcloud-compute-17   | ACTIVE | -          | Running     | ctlplane=192.0.2.45 |
| b6ed3c04-9064-4555-90ea-c1b06b27da00 | overcloud-compute-18   | ACTIVE | -          | Running     | ctlplane=192.0.2.48 |
| 1037d111-e4e1-45b0-9a29-e25c63f43d1d | overcloud-compute-19   | ACTIVE | -          | Running     | ctlplane=192.0.2.51 |
| 458c6e45-c0dc-44ed-8be8-f082034e2427 | overcloud-compute-2    | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 77b9fc2e-b0f2-4871-936d-927ff0cdd481 | overcloud-compute-3    | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| a19bb1d0-8b0c-4276-9323-68b2b07938a2 | overcloud-compute-4    | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| e44d5105-aa2e-4c5b-a43a-1b508fb75bd8 | overcloud-compute-5    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| 21851e51-c329-43ad-bdc2-403ce9c38eb2 | overcloud-compute-6    | ACTIVE | -          | Running     | ctlplane=192.0.2.23 |
| e1f24a55-0481-4a9d-844d-24bb8f792cae | overcloud-compute-7    | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| c87353f1-d60e-4cee-9e0a-706902030992 | overcloud-compute-8    | ACTIVE | -          | Running     | ctlplane=192.0.2.25 |
| 040f55f4-28a4-448f-b2ff-864d26812bb1 | overcloud-compute-9    | ACTIVE | -          | Running     | ctlplane=192.0.2.44 |
| e02db87b-8b83-4f3f-abef-67597eaed760 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
| 72d56c64-96d0-432f-bdd6-012d76c27659 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| d6efae4d-d76c-441f-93e8-48411f6e23c2 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_IN_PROGRESS | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_IN_PROGRESS | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 34651110-aea9-4665-9070-c095a1b1fe9e | overcloud  | UPDATE_COMPLETE | 2015-09-10T02:27:07Z |
+--------------------------------------+------------+-----------------+----------------------+
[stack@dell-undercloud ~]$


It proceed to delete all compute nodes deployed during the last failed exercise including those whose state is active, and return heat stack to update_complete state
Is this the workflow to recover heat stack?

Comment 5 chris alfonso 2015-09-14 16:52:09 UTC

We don't have a way to rollback to the last known state. However, if you've identified the cause of the failure and fix whatever was causing the failure then re-running the deployment command, heat will attempt to fix the failed resources.

Comment 6 Zane Bitter 2015-09-22 17:19:17 UTC

In this case you shouldn't really _need_ to use "openstack overcloud node delete" - that command exists to get rid of nodes that you know have failed (or you otherwise have your own reasons for getting rid of), but nobody has told Heat. In this case, the nodes went into the ERROR state in Nova while creating them, so Heat knows already that they're failed. Heat will just do the Right Thing on the next update (so just using the deploy command again should get it to recover) - which is to say it will create replacement resources and delete the failed ones.

That said, the "openstack overcloud node delete" command should still do the Right Thing, and it's a bug that it's not. Specifically bug 1255910 - because the client reads the current size from the parameters of the stack, and because the parameters are not currently saved on a failed update, after a failed update the next attempt to remove a node or nodes will reduce the size to the number in the most recent successful update less the number being removed. I'm inferring here that you had 20 computes, failed to scale up to 30 and then tried to delete one and ended up with 19 as a result. That seems completely consistent, so I'm marking this bug as a duplicate.

Comment 7 Zane Bitter 2015-09-22 17:19:58 UTC


*** This bug has been marked as a duplicate of bug 1255910 ***

Comment 8 Red Hat Bugzilla 2023-09-14 03:05:00 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days