Bug 1629062

Summary:	Overcloud redeploy hangs on the ServiceChain steps
Product:	Red Hat OpenStack	Reporter:	Marian Krcmarik <mkrcmari>
Component:	openstack-heat	Assignee:	Thomas Hervé <therve>
Status:	CLOSED ERRATA	QA Contact:	Gurenko Alex <agurenko>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	14.0 (Rocky)	CC:	afazekas, ahrechan, apevec, emacchi, jjoyce, jslagle, lhh, mburns, mcornea, michele, mkrcmari, ramishra, sbaker, shardy, srevivo, therve
Target Milestone:	beta	Keywords:	Triaged
Target Release:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-heat-11.0.1-0.20181010161427.46aacab.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-01-11 11:53:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marian Krcmarik 2018-09-14 18:54:03 UTC

Description of problem:
Redeploy of freshly deployed OSP14 hangs and timeouts on ServiceChain steps. This is original deployment command line:
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_62.log

The redeploy command just adds fencing templates:
-e fencing.yaml \

Version-Release number of selected component (if applicable):
$ rpm -qa | grep heat
python-heat-agent-json-file-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
python-heat-agent-docker-cmd-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
python-tripleoclient-heat-installer-10.5.1-0.20180901082351.6d7aa74.el7ost.noarch
python-heat-agent-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
openstack-heat-monolith-11.0.1-0.20180901130821.680a515.el7ost.noarch
python2-heatclient-1.16.1-0.20180810081134.b5f3d34.el7ost.noarch
python-heat-agent-ansible-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
openstack-tripleo-heat-templates-9.0.0-0.20180831204457.17bb71e.0rc1.el7ost.noarch
python-heat-agent-hiera-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
openstack-heat-api-11.0.1-0.20180901130821.680a515.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
puppet-heat-13.3.1-0.20180831195745.28088f9.el7ost.noarch
openstack-heat-common-11.0.1-0.20180901130821.680a515.el7ost.noarch
python-heat-agent-apply-config-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
openstack-heat-agents-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
python-heat-agent-puppet-1.7.1-0.20180829044839.24f9e9c.el7ost.noarch
openstack-heat-engine-11.0.1-0.20180901130821.680a515.el7ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy OSP14
2. Redeploy OSP14

Actual results:
Redeploy timeouts

Expected results:
Successful redeploy

Additional info:
Some logs to come:)

Comment 9 James Slagle 2018-09-24 20:24:10 UTC

the issue here is that Heat never frees memory after doing any stack operation. All you need to do is just stack update an overcloud and watch the Heat memory consumption infinitely grow.

Problem is solved by restarting heat_engine after each stack update.

If this is expected behavior, please let us know.

Comment 14 Rabi Mishra 2018-10-03 05:14:52 UTC

*** Bug 1635395 has been marked as a duplicate of this bug. ***

Comment 21 James Slagle 2018-10-16 21:22:12 UTC

*** Bug 1632461 has been marked as a duplicate of this bug. ***

Comment 26 Artem Hrechanychenko 2018-12-12 11:32:00 UTC

Verified
[stack@undercloud-0 ~]$ rpm -qa "*openstack-heat-*"
openstack-heat-common-11.0.1-0.20181010161427.46aacab.el7ost.noarch
openstack-heat-agents-1.7.1-0.20180907213355.476aae2.el7ost.noarch
openstack-heat-api-11.0.1-0.20181010161427.46aacab.el7ost.noarch
openstack-heat-engine-11.0.1-0.20181010161427.46aacab.el7ost.noarch
openstack-heat-monolith-11.0.1-0.20181010161427.46aacab.el7ost.noarch

Comment 28 errata-xmlrpc 2019-01-11 11:53:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045