Bug 1738922

Summary: High CPU consumption by heat even an any stack update at large stack size
Product: Red Hat OpenStack Reporter: Sai Sindhur Malleni <smalleni>
Component: instack-undercloudAssignee: Rabi Mishra <ramishra>
Status: CLOSED ERRATA QA Contact: Luke Short <lshort>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: aschultz, dpeacock, emacchi, jschluet, lshort, mburns, ramishra, sbaker, shardy
Target Milestone: z9Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)Flags: ramishra: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: instack-undercloud-8.4.7-13.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-07 14:01:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sai Sindhur Malleni 2019-08-08 11:46:18 UTC 
Description of problem:
At a stack size of 250+ nodes, any minor operation on the stack such as deleting a single node also leads to very high CPU consumption by heat-engine workers. This could lead to major problems and slowdowns, especially if the VM has not been properly sized.

https://snapshot.raintank.io/dashboard/snapshot/y8VugcvdXy6qWRYDJpaa9j6ABiflFwyr

The peaks in above graph point to scale ups/down and node deletes.
There are 25 heat-engine workers and #executor_thread_pool_size = 64

Version-Release number of selected component (if applicable):
OSP 13

How reproducible:
100%

Steps to Reproduce:
1. Deploy a large heat stack
2. Perform update on the stack like deleting a node
3.

Actual results:

heat-engine consumes upwards of 15 cores

Expected results:

CPU usage should be lower

Additional info:
Comment 17 Alex McLeod 2019-10-31 11:32:43 UTC 
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.
Comment 19 errata-xmlrpc 2019-11-07 14:01:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3794