Bug 1879418 - [RHOSP16.1][DCN][Multistack] - latest config download status reported for all stacks
Summary: [RHOSP16.1][DCN][Multistack] - latest config download status reported for all...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation
Version: 16.1 (Train)
Hardware: All
OS: All
low
low
Target Milestone: z3
: 16.1 (Train on RHEL 8.2)
Assignee: RHOS Documentation Team
QA Contact: RHOS Documentation Team
URL:
Whiteboard:
Depends On:
Blocks: 1899646
TreeView+ depends on / blocked
 
Reported: 2020-09-16 09:16 UTC by Nilesh
Modified: 2023-12-15 19:25 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
It is a known issue that the `openstack overcloud status` command might not return the correct status for a given stack name when multiple stacks exist. Instead, the status of the most recently deployed stack is always returned, regardless of the stack name. This can lead to failure reported for all stacks when it is only the most recently deployed stack that has failed. Workaround: The true status of the deployment must be clear. For example, `openstack stack list` shows any overcloud deployment failures in the heat stage and the ansible deployment logs show failures in the config download stage.
Clone Of:
: 1899646 (view as bug list)
Environment:
Last Closed: 2021-03-01 16:09:05 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1895824 0 None None None 2020-09-16 10:59:31 UTC
OpenStack gerrit 752219 0 None MERGED overcloud status report the correct cd for the given plan 2021-01-04 16:23:46 UTC
Red Hat Issue Tracker OSP-30885 0 None None None 2023-12-15 19:25:23 UTC

Description Nilesh 2020-09-16 09:16:27 UTC 
Heelo Team, 


* Upon checking the status of the stack, noticed that the **central** stack is getting touched, this should not happened. 

~~~
+--------------+-------------------+
|  Plan Name   | Deployment Status |
+--------------+-------------------+
| central |     DEPLOYING     |
+--------------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

* why central stack is touched. 

+--------------+-------------------+
|  Plan Name   | Deployment Status |
+--------------+-------------------+
| central |   DEPLOY_FAILED   |
+--------------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |   DEPLOY_FAILED   |
+-----------+-------------------+

I am using below commands to check the stack details. 

~~~
openstack overcloud status --stack <central/dcn0/dcn1>
openstack overcloud status --plan <central/dcn0/dcn1>
~~~

* https://github.com/openstack/tripleo-common/blob/stable/train/tripleo_common/actions/deployment.py#L455
Comment 1 Nilesh 2020-09-16 11:25:17 UTC 
Started redeployment: 

~~~
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status       | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_IN_PROGRESS | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE    | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |   DEPLOY_SUCCESS  |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~

* Waited for sometime, central stack is touched. 

~~~
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name           | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   | 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             | False	|
| 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   | b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             | False	|
| b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   | 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             | False	|
| 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 | 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             | False	|
| 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 | 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             | False	|
| ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 | 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             | False	|
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |     DEPLOYING     |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |     DEPLOYING     |
+-----------+-------------------+
~~~
Comment 2 Ollie Walsh 2020-09-16 11:28:26 UTC 
(In reply to Nilesh from comment #1)
> Started redeployment: 
> 
> ~~~
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> | ID                                   | Stack Name | Project               
> | Stack Status       | Creation Time        | Updated Time |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> | 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_IN_PROGRESS | 2020-09-16T11:00:01Z
> | None         |
> | 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE    | 2020-09-16T08:17:43Z
> | None         |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+--------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |  central  |   DEPLOY_SUCCESS  |
> +-----------+-------------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |    dcn0   |     DEPLOYING     |
> +-----------+-------------------+
> ~~~
> 
> * Waited for sometime, central stack is touched. 
> 
> ~~~
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> | UUID                                 | Name           | Instance UUID     
> | Power State | Provisioning State | Maintenance |
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> | a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   |
> 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             |
> False	|
> | 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   |
> b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             |
> False	|
> | b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   |
> 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             |
> False	|
> | 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 |
> 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             |
> False	|
> | 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 |
> 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             |
> False	|
> | ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 |
> 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             |
> False	|
> +--------------------------------------+----------------+--------------------
> ------------------+-------------+--------------------+-------------+
> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> | ID                                   | Stack Name | Project               
> | Stack Status    | Creation Time        | Updated Time |
> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> | 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z |
> None         |
> | 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    |
> 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z |
> None         |

Can see here ^ that the central stack is not touched

> +--------------------------------------+------------+------------------------
> ----------+-----------------+----------------------+--------------+
> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |  central  |     DEPLOYING     |
> +-----------+-------------------+

This is just reporting the latest config_download status, ignoring the plan name, so are the dcn0 deployment that is currently in progress.

> +-----------+-------------------+
> | Plan Name | Deployment Status |
> +-----------+-------------------+
> |    dcn0   |     DEPLOYING     |
> +-----------+-------------------+
> ~~~
Comment 4 Nilesh 2020-09-16 12:00:18 UTC 
* And my edge deployment failed, but dont know why central is also showing failed. 


Every 0.1s: openstack baremetal node list ; openstack stack list; openstack overcloud status --stack central ; openstack overcloud status --stack dcn0                                                                                        16dcn.example.local: Wed Sep 16 07:55:53 2020

+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name           | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
| a9d0a5a2-010c-4da3-ae21-d4e13ad9175f | controller-0   | 45fb91c9-221e-446c-9d88-4d58362fcd95 | power on    | active             | False	|
| 274a4dd9-dc6f-4ccc-b05f-77ca3255ed50 | controller-1   | b2bd3661-30cd-4eb1-9c50-166aa0b3bdb5 | power on    | active             | False	|
| b9074c93-3f1d-48f6-9c3b-54a45fb08be9 | controller-2   | 5fbb6062-3eb3-48fd-a70d-f474df254fef | power on    | active             | False	|
| 2bc1a6a5-c384-4698-8617-39ea6332e488 | dcn0-compute-0 | 2dbd417e-3ad0-4710-8b12-bb97fad9c3a2 | power on    | active             | False	|
| 30ee4426-bb73-4f4c-a928-b3c369ed4c31 | dcn0-compute-1 | 86a6bd58-3746-4976-8948-0fd627095bd2 | power on    | active             | False	|
| ba6e88f6-76be-4d6c-b07f-69021962aecc | dcn0-compute-2 | 989b265d-b9ce-44c9-84ed-fac877cf1429 | power on    | active             | False	|
+--------------------------------------+----------------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 1dbdfb32-de71-44c2-a421-faed0fdf4f71 | dcn0       | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T11:00:01Z | None         |
| 66267f47-d2b7-4d9e-9db3-ef66df48cf19 | central    | 3eaacfb8392643c6bc83944037731f6e | CREATE_COMPLETE | 2020-09-16T08:17:43Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|  central  |   DEPLOY_FAILED   |
+-----------+-------------------+
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
|    dcn0   |   DEPLOY_FAILED   |
+-----------+-------------------+
Comment 7 Ollie Walsh 2020-11-19 17:50:19 UTC 
Postponing to 16.1.4 in BZ1899646. Document as a known issue for 16.1.3
Comment 8 Dan Macpherson 2021-03-01 16:02:53 UTC 
@owalsh -- It looks like this is an engineering BZ, not a docs BZ. I think it was implemented as a Known Issue in our release notes due to the requires_doc_text flag and the Doc Text field filled out.

Should I switch this back to openstack-tripleo-common?
Comment 9 Ollie Walsh 2021-03-01 16:08:37 UTC 
(In reply to Dan Macpherson from comment #8)
> @owalsh -- It looks like this is an engineering BZ, not a docs

This BZ is just to ensure it is documented as a known issue in 16.1.z3. Engineering bug is https://bugzilla.redhat.com/show_bug.cgi?id=1899646 targeting 16.1z4.

> BZ. I think it was implemented as a Known Issue in our release notes due to
> the requires_doc_text flag and the Doc Text field filled out.

Great. I'll close this BZ.
Note You need to log in before you can comment on or make changes to this bug.