Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1637966

Summary: openstack overcloud status shows DEPLOY_SUCCESS when the stack status is CREATE_FAILED
Product: Red Hat OpenStack Reporter: Andrey Fiodorov <afiodoro>
Component: openstack-tripleo-commonAssignee: James Slagle <jslagle>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: agurenko, aschultz, gfidente, jslagle, mburns, slinaber, ukalifon
Target Milestone: betaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-9.4.1-0.20181012010869.67bab16.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:53:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1648669    
Bug Blocks:    

Description Andrey Fiodorov 2018-10-10 12:13:24 UTC
Description of problem:
Our deployment failed with error "503 Service Unavailable".
Apparently the heat service is down:
(undercloud) [stack@undercloud-0 ~]$ sudo docker ps |grep heat
2db1dff00df6        192.168.24.1:8787/rhosp14/openstack-heat-api-cfn:2018-10-08.4                "kolla_start"            5 hours ago         Up 5 hours (unhealthy)
                    heat_api_cfn
35b191b639f2        192.168.24.1:8787/rhosp14/openstack-heat-api:2018-10-08.4                    "kolla_start"            5 hours ago         Up 5 hours
                    heat_api_cron
c804c8eb85aa        192.168.24.1:8787/rhosp14/openstack-heat-engine:2018-10-08.4                 "kolla_start"            5 hours ago         Up 5 hours (healthy)
                    heat_engine
33157896efc6        192.168.24.1:8787/rhosp14/openstack-heat-api:2018-10-08.4                    "kolla_start"            5 hours ago         Up 5 hours (unhealthy)
                    heat_api
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| f1517388-13c5-48c5-b2cc-5fc7a2468502 | overcloud  | 57c16282a45444fe9c4fe49c8ae4206e | CREATE_FAILED | 2018-10-10T09:09:58Z | None         |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-10-10 09:05:58 | 2018-10-10 09:05:58 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+

And this issue confuses the GUI. The GUI is showed: Deployment success.

Version-Release number of selected component (if applicable):
openstack-tripleo-common-container-base-9.3.1-0.20180923215327.d22cb3e.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install OSP14 and deploy overcloud
2. Stop the heat service shortly after the deployment started
3.

Actual results:


Expected results:


Additional info:
SOSReport will be attached

Comment 15 Udi Kalifon 2018-11-02 12:54:52 UTC
You can also see the opposite situation, where the overcloud status (and the GUI) show FAILED and the stack status shows CREATE_COMPLETE:


(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan plan

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
|    plan   | 2018-11-02 10:50:30 | 2018-11-02 10:50:30 |   DEPLOY_FAILED   |
+-----------+---------------------+---------------------+-------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+-----------+------------+--------------+-----------------+--------------+
| ID        | Stack Name | Project      | Stack Status    | Updated Time |
+-----------+------------+--------------+-----------------+--------------+
| 3fd4ed.,. | plan       | cf304e09d... | CREATE_COMPLETE | None         |
+-----------+------------+--------------+-----------------+--------------+

Comment 16 Udi Kalifon 2018-11-02 13:07:25 UTC
Workaround (see also bug 1644861):
1) openstack object delete <<plan-name>>-messages deployment_status.yaml
2) F5 in the GUI
3) Click on "recover deployment status"

Comment 19 Beth White 2018-11-13 13:28:50 UTC
*** Bug 1637947 has been marked as a duplicate of this bug. ***

Comment 20 Gurenko Alex 2018-11-15 10:50:31 UTC
Verified on puddle 2018-11-13.1

[stack@undercloud-0 ~]$ rpm -q openstack-tripleo-common
openstack-tripleo-common-9.4.1-0.20181012010875.67bab16.el7ost.noarch

Comment 22 errata-xmlrpc 2019-01-11 11:53:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045