Bug 1738922 - High CPU consumption by heat even an any stack update at large stack size
Summary: High CPU consumption by heat even an any stack update at large stack size
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z9
: 13.0 (Queens)
Assignee: Rabi Mishra
QA Contact: Luke Short
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-08 11:46 UTC by Sai Sindhur Malleni
Modified: 2019-11-07 14:02 UTC (History)
9 users (show)

Fixed In Version: instack-undercloud-8.4.7-13.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-07 14:01:46 UTC
Target Upstream Version:
Embargoed:
ramishra: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3794 0 None None None 2019-11-07 14:02:01 UTC

Description Sai Sindhur Malleni 2019-08-08 11:46:18 UTC
Description of problem:
At a stack size of 250+ nodes, any minor operation on the stack such as deleting a single node also leads to very high CPU consumption by heat-engine workers. This could lead to major problems and slowdowns, especially if the VM has not been properly sized.

https://snapshot.raintank.io/dashboard/snapshot/y8VugcvdXy6qWRYDJpaa9j6ABiflFwyr

The peaks in above graph point to scale ups/down and node deletes.
There are 25 heat-engine workers and #executor_thread_pool_size = 64

Version-Release number of selected component (if applicable):
OSP 13

How reproducible:
100%

Steps to Reproduce:
1. Deploy a large heat stack
2. Perform update on the stack like deleting a node
3.

Actual results:

heat-engine consumes upwards of 15 cores

Expected results:

CPU usage should be lower

Additional info:

Comment 17 Alex McLeod 2019-10-31 11:32:43 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 19 errata-xmlrpc 2019-11-07 14:01:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3794


Note You need to log in before you can comment on or make changes to this bug.