Bug 1463506

Summary: High CPU load and slow response times on the smart proxy (smart_proxy_dynflow_core process) with rex at scale
Product: Red Hat Satellite Reporter: Ivan Necas <inecas>
Component: Remote ExecutionAssignee: Ivan Necas <inecas>
Status: CLOSED ERRATA QA Contact: Jan Hutař <jhutar>
Severity: high Docs Contact:
Priority: high    
Version: 6.3.0CC: aruzicka, bbuckingham, bkearney, cdonnell, inecas, jcallaha, jhutar, oshtaier
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygems-foreman-tasks-core-0.1.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1487964 (view as bug list) Environment:
Last Closed: 2018-02-21 17:06:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1487964    
Attachments:
Description Flags
Patch for more effective refreshing of remote execution commands none

Description Ivan Necas 2017-06-21 07:17:53 UTC 
Created attachment 1289960 [details]
Patch for  more effective refreshing of remote execution commands

Description of problem:
Due to inefficiency of the refreshing of execution, smart_proxy_dynflow_core can
get under heavy CPU load, which can eventually lead to slow response times
and bloated queues.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. run remote execution job against 1000 hosts


Actual results:
smart_proxy_dynflow_core getting to 100% CPU soon, response times
on smart_proxy getting to tens of seconds

Expected results:
Moderate CPU load, fast reponse times on smart-proxy (below second)

Additional info:
Extracted from https://bugzilla.redhat.com/show_bug.cgi?id=1416542#c36 to have more clarity on different parts of the bug.
Comment 4 Adam Ruzicka 2017-07-03 07:25:37 UTC 
Created redmine issue http://projects.theforeman.org/issues/20182 from this bug
Comment 5 Brad Buckingham 2017-07-04 22:24:26 UTC 
Moving to ASSIGNED as there is a PR submitted upstream.
Comment 6 Satellite Program 2017-07-10 16:12:32 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20182 has been resolved.
Comment 13 Bryan Kearney 2018-02-21 17:06:18 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336