Bug 1879418

Summary:	[RHOSP16.1][DCN][Multistack] - latest config download status reported for all stacks
Product:	Red Hat OpenStack	Reporter:	Nilesh <nchandek>
Component:	documentation	Assignee:	RHOS Documentation Team <rhos-docs>
Status:	CLOSED DEFERRED	QA Contact:	RHOS Documentation Team <rhos-docs>
Severity:	low	Docs Contact:
Priority:	low
Version:	16.1 (Train)	CC:	bdobreli, dmacpher, joflynn, mburns, ndeevy, owalsh, slinaber
Target Milestone:	z3	Keywords:	Documentation, Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	It is a known issue that the `openstack overcloud status` command might not return the correct status for a given stack name when multiple stacks exist. Instead, the status of the most recently deployed stack is always returned, regardless of the stack name. This can lead to failure reported for all stacks when it is only the most recently deployed stack that has failed. Workaround: The true status of the deployment must be clear. For example, `openstack stack list` shows any overcloud deployment failures in the heat stage and the ansible deployment logs show failures in the config download stage.	Story Points:	---
Clone Of:
Clones:	1899646 (view as bug list)		Environment:
Last Closed:	2021-03-01 16:09:05 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1899646

Description Nilesh 2020-09-16 09:16:27 UTC

Heelo Team, 


* Upon checking the status of the stack, noticed that the **central** stack is getting touched, this should not happened. 

~~~
+--------------+-------------------+
|  Plan Name   | Deployment Status |
+--------------+-------------------+
| central |     DEPLOYING     |
+--------------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

* why central stack is touched. 

+--------------+-------------------+
|  Plan Name   | Deployment Status |
+--------------+-------------------+
| central |   DEPLOY_FAILED   |
+--------------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |   DEPLOY_FAILED   |
+-----------+-------------------+

I am using below commands to check the stack details. 

~~~
openstack overcloud status --stack <central/dcn0/dcn1>
openstack overcloud status --plan <central/dcn0/dcn1>
~~~

* https://github.com/openstack/tripleo-common/blob/stable/train/tripleo_common/actions/deployment.py#L455

Comment 1 Nilesh 2020-09-16 11:25:17 UTC

Started redeployment: 

~~~
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status       | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_IN_PROGRESS | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE    | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |   DEPLOY_SUCCESS  |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

* Waited for sometime, central stack is touched. 

~~~
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name           | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   | 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             | False	|
| 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   | b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             | False	|
| b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   | 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             | False	|
| 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 | 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             | False	|
| 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 | 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             | False	|
| ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 | 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             | False	|
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |     DEPLOYING     |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

Comment 2 Ollie Walsh 2020-09-16 11:28:26 UTC

(In reply to Nilesh from comment #1)
> Started redeployment: 
> 
> ~~~
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> | ID                                   | Stack Name | Project               
> | Stack Status       | Creation Time        | Updated Time |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> | 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_IN_PROGRESS | 2020-09-16T11:00:01Z
> | None         |
> | 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE    | 2020-09-16T08:17:43Z
> | None         |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |  central  |   DEPLOY_SUCCESS  |
> +-----------+-------------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |    dcn0   |     DEPLOYING     |
> +-----------+-------------------+
> ~~~
> 
> * Waited for sometime, central stack is touched. 
> 
> ~~~
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> | UUID                                 | Name           | Instance UUID     
> | Power State | Provisioning State | Maintenance |
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> | a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   |
> 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             |
> False	|
> | 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   |
> b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             |
> False	|
> | b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   |
> 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             |
> False	|
> | 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 |
> 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             |
> False	|
> | 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 |
> 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             |
> False	|
> | ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 |
> 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             |
> False	|
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> | ID                                   | Stack Name | Project               
> | Stack Status    | Creation Time        | Updated Time |
> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> | 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z |
> None         |
> | 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z |
> None         |

Can see here ^ that the central stack is not touched

> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |  central  |     DEPLOYING     |
> +-----------+-------------------+

This is just reporting the latest config_download status, ignoring the plan name, so are the dcn0 deployment that is currently in progress.

> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |    dcn0   |     DEPLOYING     |
> +-----------+-------------------+
> ~~~

Comment 4 Nilesh 2020-09-16 12:00:18 UTC

* And my edge deployment failed, but dont know why central is also showing failed. 


Every 0.1s: openstack baremetal node list ; openstack stack list; openstack overcloud status --stack central ; openstack overcloud status --stack dcn0                                                                                        16dcn.example.local: Wed Sep 16 07:55:53 2020

+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name           | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   | 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             | False	|
| 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   | b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             | False	|
| b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   | 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             | False	|
| 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 | 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             | False	|
| 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 | 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             | False	|
| ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 | 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             | False	|
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |   DEPLOY_FAILED   |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |   DEPLOY_FAILED   |
+-----------+-------------------+

Comment 7 Ollie Walsh 2020-11-19 17:50:19 UTC

Postponing to 16.1.4 in BZ1899646. Document as a known issue for 16.1.3

Comment 8 Dan Macpherson 2021-03-01 16:02:53 UTC

@owalsh -- It looks like this is an engineering BZ, not a docs BZ. I think it was implemented as a Known Issue in our release notes due to the requires_doc_text flag and the Doc Text field filled out.

Should I switch this back to openstack-tripleo-common?

Comment 9 Ollie Walsh 2021-03-01 16:08:37 UTC

(In reply to Dan Macpherson from comment #8)
> @owalsh -- It looks like this is an engineering BZ, not a docs

This BZ is just to ensure it is documented as a known issue in 16.1.z3. Engineering bug is https://bugzilla.redhat.com/show_bug.cgi?id=1899646 targeting 16.1z4.

> BZ. I think it was implemented as a Known Issue in our release notes due to
> the requires_doc_text flag and the Doc Text field filled out.

Great. I'll close this BZ.