Bug 1463506 - High CPU load and slow response times on the smart proxy (smart_proxy_dynflow_core process) with rex at scale
High CPU load and slow response times on the smart proxy (smart_proxy_dynflow...
Status: ON_QA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Remote Execution (Show other bugs)
6.3.0
Unspecified Unspecified
high Severity high (vote)
: GA
: --
Assigned To: Ivan Necas
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-21 03:17 EDT by Ivan Necas
Modified: 2017-08-15 16:33 EDT (History)
7 users (show)

See Also:
Fixed In Version: tfm-rubygems-foreman-tasks-core-0.1.4
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch for more effective refreshing of remote execution commands (4.78 KB, patch)
2017-06-21 03:17 EDT, Ivan Necas
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 20182 None None None 2017-07-03 03:25 EDT

  None (edit)
Description Ivan Necas 2017-06-21 03:17:53 EDT
Created attachment 1289960 [details]
Patch for  more effective refreshing of remote execution commands

Description of problem:
Due to inefficiency of the refreshing of execution, smart_proxy_dynflow_core can
get under heavy CPU load, which can eventually lead to slow response times
and bloated queues.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. run remote execution job against 1000 hosts


Actual results:
smart_proxy_dynflow_core getting to 100% CPU soon, response times
on smart_proxy getting to tens of seconds

Expected results:
Moderate CPU load, fast reponse times on smart-proxy (below second)

Additional info:
Extracted from https://bugzilla.redhat.com/show_bug.cgi?id=1416542#c36 to have more clarity on different parts of the bug.
Comment 4 aruzicka 2017-07-03 03:25:37 EDT
Created redmine issue http://projects.theforeman.org/issues/20182 from this bug
Comment 5 Brad Buckingham 2017-07-04 18:24:26 EDT
Moving to ASSIGNED as there is a PR submitted upstream.
Comment 6 pm-sat@redhat.com 2017-07-10 12:12:32 EDT
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20182 has been resolved.

Note You need to log in before you can comment on or make changes to this bug.