Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1624091

Summary:

Need to have a quick and reliable way to see the status of the last OC deployment.

Product:

Red Hat OpenStack

Reporter:

Alexander Chuzhoy <sasha>

Component:

rhosp-director

Assignee:

James Slagle <jslagle>

Status:

CLOSED INSUFFICIENT_DATA

QA Contact:

Gurenko Alex <agurenko>

Severity:

medium

Docs Contact:

Priority:

high

Version:

14.0 (Rocky)

CC:

aschultz, dbecker, dmacpher, dpeacock, jrouleau, jslagle, mburns, mlammon, morazi, ohochman, sasha, srevivo

Target Milestone:

---

Keywords:

Triaged

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-10-01 21:45:12 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1615694

Attachments:

Description	Flags
heat and mistral logs	none

Description Alexander Chuzhoy 2018-08-30 22:37:11 UTC

Need to have a quick and reliable way to see the status of the last OC deployment.
Environment:
instack-undercloud-9.2.1-0.20180803181448.be5fa97.el7ost.noarch                                                                                                                  
openstack-tripleo-heat-templates-9.0.0-0.20180804083746.el7ost.noarch 
python2-mistralclient-3.7.0-0.20180731011525.f0ee48f.el7ost.noarch                                                                                                               
python2-mistral-lib-1.0.0-0.20180730234322.d1ccfd0.el7ost.noarch                                                                                                                 
puppet-mistral-13.2.0-0.20180731214249.7dd767e.el7ost.noarch    


Using "openstack stack list" doesn't reflect the status of OC deployment anymore.
We need to have an option to check the status of the last deployment.

Comment 2 Jill Rouleau 2018-08-30 23:38:15 UTC

The new 'openstack overcloud status' command helps with this.

https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ansible_config_download.html#deployment-status

Comment 4 Alexander Chuzhoy 2018-08-31 01:37:50 UTC

Let's make sure it's documented:
openstack overcloud status --plan <plan name>

Comment 5 Dan Macpherson 2018-08-31 06:55:32 UTC

This is related to BZ#1615694. I'll add this as a dependency.

Comment 6 Alexander Chuzhoy 2018-09-12 20:23:45 UTC

Seems like relying on "openstack overcloud status --plan <plan name>" for checking the status of the deployment isn't right:


Ran the same command twice on the same machine with 1 minute interval:

(undercloud) [stack@undercloud75 ~]$ openstack overcloud status --plan overcloud

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-12 19:40:25 | 2018-09-12 19:48:25 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+

(undercloud) [stack@undercloud75 ~]$ openstack overcloud status --plan overcloud

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-12 20:17:31 | 2018-09-12 20:19:44 |     DEPLOYING     |
+-----------+---------------------+---------------------+-------------------+

Comment 7 Omri Hochman 2018-09-12 20:30:43 UTC

(In reply to Alexander Chuzhoy from comment #6)
> Seems like relying on "openstack overcloud status --plan <plan name>" for
> checking the status of the deployment isn't right:
> 

If the status switch from DEPLOY_SUCCESS to DEPLOYING and back to DEPLOY_SUCCESS

I wonder if we can somehow have it work: 
DEPLOYING (when it's deploying..) -->  DEPLOY_SUCCESS  (when it's really done deploying.. ) 


Not sure what do we want to document on this.

Comment 8 Alexander Chuzhoy 2018-09-12 20:47:48 UTC

Moving from documentation , because it seems like there's no way to see the status of overcloud deployment reliably.

Comment 9 James Slagle 2018-09-19 18:53:17 UTC

please provide specific instructions to reproduce. the output shown in comment 6 doesn't line up with the description, unless you're saying the timestamps jumped 30+ minutes into the future.

Comment 10 Alexander Chuzhoy 2018-09-27 13:46:21 UTC

To reproduce:

delete the overcloud stack and compare the output from:
"openstack stack list"
and
"openstack overcloud status --plan overcloud"


(undercloud) [stack@undercloud-0 ~]$ date;  openstack overcloud status --plan overcloud
Thu Sep 27 08:32:51 EDT 2018

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 03:36:35 | 2018-09-27 04:34:47 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+


(undercloud) [stack@undercloud-0 ~]$ openstack stack delete overcloud -y


(undercloud) [stack@undercloud-0 ~]$ date; openstack stack list
Thu Sep 27 08:35:56 EDT 2018
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
| ID                                   | Stack Name | Project                          | Stack Status       | Creation Time        | Updated Time         |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
| 2304a62c-ab8e-4c5b-8ab6-f73678ca3084 | overcloud  | c054c77ed97f4311b858f048fe8d280a | DELETE_IN_PROGRESS | 2018-09-27T03:01:38Z | 2018-09-27T12:33:15Z |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
(undercloud) [stack@undercloud-0 ~]$ date;  openstack overcloud status --plan overcloud
Thu Sep 27 08:36:11 EDT 2018

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 03:36:35 | 2018-09-27 04:34:47 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+





Few more minutes into deletion:
(undercloud) [stack@undercloud-0 ~]$ date; openstack stack list
Thu Sep 27 08:44:10 EDT 2018
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
| ID                                   | Stack Name | Project                          | Stack Status       | Creation Time        | Updated Time         |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
| 2304a62c-ab8e-4c5b-8ab6-f73678ca3084 | overcloud  | c054c77ed97f4311b858f048fe8d280a | DELETE_IN_PROGRESS | 2018-09-27T03:01:38Z | 2018-09-27T12:33:15Z |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
(undercloud) [stack@undercloud-0 ~]$ date;  openstack overcloud status --plan overcloud
Thu Sep 27 08:44:17 EDT 2018

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 03:36:35 | 2018-09-27 04:34:47 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+

After deletion:
(undercloud) [stack@undercloud-0 ~]$ date; openstack stack list
Thu Sep 27 09:00:25 EDT 2018

(undercloud) [stack@undercloud-0 ~]$ date;  openstack overcloud status --plan overcloud
Thu Sep 27 09:00:33 EDT 2018

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 03:36:35 | 2018-09-27 04:34:47 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+


### and somewhat later...
(undercloud) [stack@undercloud-0 ~]$ date;  openstack overcloud status --plan overcloud
Thu Sep 27 09:40:00 EDT 2018

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 03:36:35 | 2018-09-27 04:34:47 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+


Started a new deployment and finally the status switched to
+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 13:43:40 | 2018-09-27 13:43:40 |     DEPLOYING     |
+-----------+---------------------+---------------------+-------------------+

Comment 11 Alexander Chuzhoy 2018-09-27 15:47:47 UTC

Trying to delete overcloud with 'openstack overcloud delete overcloud' instead:

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud delete overcloud
Are you sure you want to delete this overcloud [y/N]? y
Deleting stack overcloud...


Waiting for messages on queue 'tripleo' with no timeout.



Error occurred during stack delete None
(undercloud) [stack@undercloud-0 ~]$ 

(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
| ID                                   | Stack Name | Project                          | Stack Status       | Creation Time        | Updated Time         |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
| c7561cb9-9722-46f9-9079-d34188e7fb90 | overcloud  | 5e0ec8d1cd1148c984306c3f8a783ed9 | DELETE_IN_PROGRESS | 2018-09-26T22:42:42Z | 2018-09-27T15:32:29Z |
+--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan overcloud
None
None


....stuck

Comment 12 Alexander Chuzhoy 2018-09-27 15:52:12 UTC

We probably should instruct not to use 'openstack stack list' at all.
Different result:
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| 3e91480f-a5bc-4fbd-b233-7ee6d288d365 | overcloud  | c054c77ed97f4311b858f048fe8d280a | CREATE_FAILED | 2018-09-27T13:48:46Z | None         |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan overcloud

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-09-27 13:43:40 | 2018-09-27 13:48:48 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+

Comment 13 James Slagle 2018-10-01 12:07:46 UTC

(In reply to Alexander Chuzhoy from comment #11)
> Trying to delete overcloud with 'openstack overcloud delete overcloud'
> instead:
> 
> (undercloud) [stack@undercloud-0 ~]$ openstack overcloud delete overcloud
> Are you sure you want to delete this overcloud [y/N]? y
> Deleting stack overcloud...
> 
> 
> Waiting for messages on queue 'tripleo' with no timeout.
> 
> 
> 
> Error occurred during stack delete None
> (undercloud) [stack@undercloud-0 ~]$ 
> 
> (undercloud) [stack@undercloud-0 ~]$ openstack stack list
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+----------------------
> +
> | ID                                   | Stack Name | Project               
> | Stack Status       | Creation Time        | Updated Time         |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+----------------------
> +
> | c7561cb9-9722-46f9-9079-d34188e7fb90 | overcloud  |
> 5e0ec8d1cd1148c984306c3f8a783ed9 | DELETE_IN_PROGRESS | 2018-09-26T22:42:42Z
> | 2018-09-27T15:32:29Z |
> +--------------------------------------+------------+------------------------
> ----------+--------------------+----------------------+----------------------
> +
> (undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan
> overcloud
> None
> None
> 
> 
> ....stuck

we can't tell how the delete failed without logs.

Please always attach or make logs available.

Comment 14 James Slagle 2018-10-01 12:08:27 UTC

removing the blocker flag. no reason to think this is a blocker atm.

Comment 15 Alexander Chuzhoy 2018-10-01 18:37:32 UTC

Providing mistral/heat logs for this deviation:
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| 7d6eee3d-a6bb-4030-bca9-d1e3decbab7f | overcloud  | 4128b1a12244434bac78ca6a83fd787b | CREATE_FAILED | 2018-10-01T17:39:52Z | None         |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan overcloud

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-10-01 17:34:16 | 2018-10-01 17:34:16 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+
(undercloud) [stack@undercloud-0 ~]$

Comment 17 Alexander Chuzhoy 2018-10-01 18:46:28 UTC

Created attachment 1489144 [details]
heat and mistral logs

Comment 18 James Slagle 2018-10-01 21:12:26 UTC

(In reply to James Slagle from comment #13)
> (In reply to Alexander Chuzhoy from comment #11)
> > Trying to delete overcloud with 'openstack overcloud delete overcloud'
> > instead:
> > 
> > (undercloud) [stack@undercloud-0 ~]$ openstack overcloud delete overcloud
> > Are you sure you want to delete this overcloud [y/N]? y
> > Deleting stack overcloud...
> > 
> > 
> > Waiting for messages on queue 'tripleo' with no timeout.
> > 
> > 
> > 
> > Error occurred during stack delete None
> > (undercloud) [stack@undercloud-0 ~]$ 
> > 
> > (undercloud) [stack@undercloud-0 ~]$ openstack stack list
> > +--------------------------------------+------------+------------------------
> > ----------+--------------------+----------------------+----------------------
> > +
> > | ID                                   | Stack Name | Project               
> > | Stack Status       | Creation Time        | Updated Time         |
> > +--------------------------------------+------------+------------------------
> > ----------+--------------------+----------------------+----------------------
> > +
> > | c7561cb9-9722-46f9-9079-d34188e7fb90 | overcloud  |
> > 5e0ec8d1cd1148c984306c3f8a783ed9 | DELETE_IN_PROGRESS | 2018-09-26T22:42:42Z
> > | 2018-09-27T15:32:29Z |
> > +--------------------------------------+------------+------------------------
> > ----------+--------------------+----------------------+----------------------
> > +
> > (undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan
> > overcloud
> > None
> > None
> > 
> > 
> > ....stuck
> 
> we can't tell how the delete failed without logs.
> 
> Please always attach or make logs available.

the logs you attached are not for this failure. please file new bugs for new issues with logs so that we can tell why things failed.

Comment 19 James Slagle 2018-10-01 21:45:12 UTC

Please file new bugs as there's about 4 different things going on in this bz.

per the description "Need to have a quick and reliable way to see the status of the last OC deployment.", that exists and is "openstack overcloud status"

If you observe a problem with that command, please file new bugs with captured output and logs so we can debug and fix the issues.

Comment 20 Alexander Chuzhoy 2018-10-15 12:27:21 UTC

Filed separate bugs.
Clearing the needinfo.