1879418 – [RHOSP16.1][DCN][Multistack] - latest config download status reported for all stacks

Bug 1879418 - [RHOSP16.1][DCN][Multistack] - latest config download status reported for all stacks

Summary: [RHOSP16.1][DCN][Multistack] - latest config download status reported for all...

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	documentation
Sub Component:
Version:	16.1 (Train)
Hardware:	All
OS:	All
Priority:	low
Severity:	low
Target Milestone:	z3
Target Release:	16.1 (Train on RHEL 8.2)
Assignee:	RHOS Documentation Team
QA Contact:	RHOS Documentation Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1899646
TreeView+	depends on / blocked

Reported:	2020-09-16 09:16 UTC by Nilesh
Modified:	2021-03-01 16:09 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	It is a known issue that the `openstack overcloud status` command might not return the correct status for a given stack name when multiple stacks exist. Instead, the status of the most recently deployed stack is always returned, regardless of the stack name. This can lead to failure reported for all stacks when it is only the most recently deployed stack that has failed. Workaround: The true status of the deployment must be clear. For example, `openstack stack list` shows any overcloud deployment failures in the heat stage and the ansible deployment logs show failures in the config download stage.
Clone Of:
Clones:	1899646 (view as bug list)
Environment:
Last Closed:	2021-03-01 16:09:05 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1895824	0	None	None	None	2020-09-16 10:59:31 UTC
OpenStack gerrit	752219	0	None	MERGED	overcloud status report the correct cd for the given plan	2021-01-04 16:23:46 UTC

Description Nilesh 2020-09-16 09:16:27 UTC

Heelo Team, 


* Upon checking the status of the stack, noticed that the **central** stack is getting touched, this should not happened. 

~~~
+--------------+-------------------+
|  Plan Name   | Deployment Status |
+--------------+-------------------+
| central |     DEPLOYING     |
+--------------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

* why central stack is touched. 

+--------------+-------------------+
|  Plan Name   | Deployment Status |
+--------------+-------------------+
| central |   DEPLOY_FAILED   |
+--------------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |   DEPLOY_FAILED   |
+-----------+-------------------+

I am using below commands to check the stack details. 

~~~
openstack overcloud status --stack <central/dcn0/dcn1>
openstack overcloud status --plan <central/dcn0/dcn1>
~~~

* https://github.com/openstack/tripleo-common/blob/stable/train/tripleo_common/actions/deployment.py#L455

Comment 1 Nilesh 2020-09-16 11:25:17 UTC

Started redeployment: 

~~~
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status       | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_IN_PROGRESS | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE    | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |   DEPLOY_SUCCESS  |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

* Waited for sometime, central stack is touched. 

~~~
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name           | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   | 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             | False	|
| 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   | b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             | False	|
| b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   | 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             | False	|
| 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 | 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             | False	|
| 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 | 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             | False	|
| ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 | 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             | False	|
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |     DEPLOYING     |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

Comment 2 Ollie Walsh 2020-09-16 11:28:26 UTC

(In reply to Nilesh from comment #1)
> Started redeployment: 
> 
> ~~~
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> | ID                                   | Stack Name | Project               
> | Stack Status       | Creation Time        | Updated Time |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> | 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_IN_PROGRESS | 2020-09-16T11:00:01Z
> | None         |
> | 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE    | 2020-09-16T08:17:43Z
> | None         |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |  central  |   DEPLOY_SUCCESS  |
> +-----------+-------------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |    dcn0   |     DEPLOYING     |
> +-----------+-------------------+
> ~~~
> 
> * Waited for sometime, central stack is touched. 
> 
> ~~~
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> | UUID                                 | Name           | Instance UUID     
> | Power State | Provisioning State | Maintenance |
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> | a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   |
> 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             |
> False	|
> | 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   |
> b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             |
> False	|
> | b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   |
> 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             |
> False	|
> | 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 |
> 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             |
> False	|
> | 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 |
> 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             |
> False	|
> | ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 |
> 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             |
> False	|
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> | ID                                   | Stack Name | Project               
> | Stack Status    | Creation Time        | Updated Time |
> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> | 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z |
> None         |
> | 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z |
> None         |

Can see here ^ that the central stack is not touched

> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |  central  |     DEPLOYING     |
> +-----------+-------------------+

This is just reporting the latest config_download status, ignoring the plan name, so are the dcn0 deployment that is currently in progress.

> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |    dcn0   |     DEPLOYING     |
> +-----------+-------------------+
> ~~~

Comment 4 Nilesh 2020-09-16 12:00:18 UTC

* And my edge deployment failed, but dont know why central is also showing failed. 


Every 0.1s: openstack baremetal node list ; openstack stack list; openstack overcloud status --stack central ; openstack overcloud status --stack dcn0                                                                                        16dcn.example.local: Wed Sep 16 07:55:53 2020

+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name           | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   | 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             | False	|
| 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   | b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             | False	|
| b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   | 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             | False	|
| 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 | 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             | False	|
| 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 | 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             | False	|
| ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 | 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             | False	|
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |   DEPLOY_FAILED   |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |   DEPLOY_FAILED   |
+-----------+-------------------+

Comment 7 Ollie Walsh 2020-11-19 17:50:19 UTC

Postponing to 16.1.4 in BZ1899646. Document as a known issue for 16.1.3

Comment 8 Dan Macpherson 2021-03-01 16:02:53 UTC

@owalsh@redhat.com -- It looks like this is an engineering BZ, not a docs BZ. I think it was implemented as a Known Issue in our release notes due to the requires_doc_text flag and the Doc Text field filled out.

Should I switch this back to openstack-tripleo-common?

Comment 9 Ollie Walsh 2021-03-01 16:08:37 UTC

(In reply to Dan Macpherson from comment #8)
> @owalsh@redhat.com -- It looks like this is an engineering BZ, not a docs

This BZ is just to ensure it is documented as a known issue in 16.1.z3. Engineering bug is https://bugzilla.redhat.com/show_bug.cgi?id=1899646 targeting 16.1z4.

> BZ. I think it was implemented as a Known Issue in our release notes due to
> the requires_doc_text flag and the Doc Text field filled out.

Great. I'll close this BZ.

Note You need to log in before you can comment on or make changes to this bug.