Bug 1691416

Summary:	Delays when many clients upload tracer data simultaneously
Product:	Red Hat Satellite	Reporter:	Brian Smith <briasmit>
Component:	katello-tracer	Assignee:	Jonathon Turel <jturel>
Status:	CLOSED ERRATA	QA Contact:	Imaan <ikaur>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.3.5	CC:	arsingh, egolov, jhutar, jturel
Target Milestone:	6.8.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-3.16.0-0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 12:58:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1869812
Bug Blocks:

Description Brian Smith 2019-03-21 14:54:31 UTC

Description of problem:
When customer patches around 60 servers simultaneously from Satellite, many of them get to the "Uploading Tracer Profile" step around the same time, and some clients have delays of over 15 minutes.  It is also possible to reproduce this by running a remote execution "katello-tracer-upload" command on multiple hosts.  


Version-Release number of selected component (if applicable):  Satellite 6.3.5 


How reproducible:  Consistently


Steps to Reproduce:
1.  Have around 160 Satellite clients with katello-host-tools-tracer installed.
2.  Create a Remote Execution job to run on the 160 clients and run the command "time katello-tracer-upload", with a 15 minute time out.


Actual results:  In customers environment, 3 of the servers timed out after 15 minutes, and 2 servers ran for over 5 minutes.  


Expected results:  Servers not taking over 15 minutes to run katello-host-tools-tracer


Additional info:
Customer ran the same Remote execution job, but set the concurrency level to 1, so that it only ran on one server at a time, and did not have any failures or delays.   

See also:  https://community.theforeman.org/t/katello-tracer-troubleshoot/8676

Comment 3 Jonathon Turel 2020-03-24 01:17:34 UTC

Created redmine issue https://projects.theforeman.org/issues/29397 from this bug

Comment 4 Bryan Kearney 2020-03-24 02:01:41 UTC

Upstream bug assigned to jturel

Comment 5 Bryan Kearney 2020-03-24 02:01:43 UTC

Upstream bug assigned to jturel

Comment 6 Bryan Kearney 2020-03-26 16:02:19 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/29397 has been resolved.

Comment 14 errata-xmlrpc 2020-10-27 12:58:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366