Bug 1737098

Summary: OSP deployment stuck for hours, then fail
Product: Red Hat OpenStack Reporter: Rabi Mishra <ramishra>
Component: openstack-heatAssignee: Rabi Mishra <ramishra>
Status: CLOSED ERRATA QA Contact: Jad Haj Yahya <jhajyahy>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: bshephar, dhill, jraju, jveiraca, ldenny, mburns, pkundal, pmannidi, sbaker, shardy, ykulkarn
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-heat-10.0.3-6.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-03 16:53:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rabi Mishra 2019-08-02 15:39:39 UTC
Description of problem:




Replacement of ControllerDl360g10V1Config (OS::Heat::StructuredConfig) resources are normal during an stack update, when the config changes. But looks like the replacement did not happen properly as after creating a resource in db with no physical_resource_id[1], the rpc call to create the configs did not happen. 

[1] https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L384

from the logs:

2019-07-21 23:12:43.137 15722 INFO heat.engine.check_resource [req-b4239f8a-f100-4007-a8c3-93ca238368e3 - admin - default default] Replacing resource with new id 9300
2019-07-21 23:12:46.870 15722 INFO heat.engine.check_resource [req-b4239f8a-f100-4007-a8c3-93ca238368e3 - admin - default default] Replacing resource with new id 9301

And the relevant resources in heat db (see no physical resource_id for the replacement resources)

MariaDB [heat]> select id, nova_instance as physical_resource_id, name, action, status, replaces from resource where id in (9300, 9301);
+------+----------------------+----------------------------+--------+----------+----------+
| id   | physical_resource_id | name                       | action | status   | replaces |
+------+----------------------+----------------------------+--------+----------+----------+
| 9300 | NULL                 | ControllerDl360g10V1Config | DELETE | COMPLETE |     9297 |
| 9301 | NULL                 | ControllerDl360g10V1Config | DELETE | COMPLETE |     9296 |
+------+----------------------+----------------------------+--------+----------+----------+
2 rows in set (0.001 sec)

MariaDB [heat]> select id, nova_instance as physical_resource_id, name, action, status, replaced_by from resource where id in (9296, 9297);
+------+--------------------------------------+----------------------------+--------+----------+-------------+
| id   | physical_resource_id                 | name                       | action | status   | replaced_by |
+------+--------------------------------------+----------------------------+--------+----------+-------------+
| 9296 | 4e24e2a1-9843-488a-bb01-53c957d37c47 | ControllerDl360g10V1Config | UPDATE | COMPLETE |        9301 |
| 9297 | 66123571-7b14-4e21-8475-69b5f99269e5 | ControllerDl360g10V1Config | UPDATE | COMPLETE |        9300 |
+------+--------------------------------------+----------------------------+--------+----------+-------------+

Comment 9 errata-xmlrpc 2019-09-03 16:53:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2625

Comment 10 Rabi Mishra 2019-12-17 16:44:46 UTC
*** Bug 1781482 has been marked as a duplicate of this bug. ***

Comment 11 Rabi Mishra 2019-12-19 16:12:00 UTC
*** Bug 1784038 has been marked as a duplicate of this bug. ***

Comment 12 PURANDHAR SAIRAM MANNIDI 2020-02-07 06:42:40 UTC
*** Bug 1800433 has been marked as a duplicate of this bug. ***

Comment 13 Rabi Mishra 2020-02-11 15:11:59 UTC
*** Bug 1796839 has been marked as a duplicate of this bug. ***

Comment 14 PURANDHAR SAIRAM MANNIDI 2020-02-25 16:00:28 UTC
*** Bug 1805648 has been marked as a duplicate of this bug. ***