Bug 1688840

Summary:	Candlepin connection times out on large virt-who checkins
Product:	Red Hat Satellite	Reporter:	jcallaha
Component:	Candlepin	Assignee:	Barnaby Court <bcourt>
Status:	CLOSED ERRATA	QA Contact:	jcallaha
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.5.0	CC:	bbuckingham, bcourt, csnyder, jsherril, khowell, wpoteat
Target Milestone:	6.5.0	Keywords:	Regression, Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-14 12:40:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description jcallaha 2019-03-14 14:59:21 UTC

Description of problem:
When submitting a large number of hypervisors to the candlepin event, either via virt-who or manually, the Hypervisors task will almost always fail due to:

Katello::Resources::Candlepin::Consumer: Timed out connecting to server  (GET /candlepin/consumers/b709bdb4-bfc9-4ad8-9841-9f1c49fb77cd)

Version-Release number of selected component (if applicable):
Satellite 6.5.0 Snap 19

How reproducible:
Almost always

Steps to Reproduce:
1. docker run --rm -e "SATHOST=$(hostname)" -e "HYPERVISORS=5000" -e "GUESTS=1" jacobcallahan/genvirt


Actual results:
The task times out with the error above

Expected results:
The tasks process all data and completes

Comment 16 Justin Sherrill 2019-03-25 16:54:52 UTC

This sounds a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1687190

can you reproduce with the newest snap (21 or newer)?

Comment 17 jcallaha 2019-03-25 20:27:52 UTC

Justin, with the upgrade to snap 21, I'm getting something interesting.

The task on in Katello now passes at 5k hypervisors (and in about an hour), but I get this error message from the initial data submittion:

{"displayMessage":"The time waiting for task 0fc23af2-a13d-42ba-a11a-c7e21ca61d3f to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)","errors":["The time waiting for task 0fc23af2-a13d-42ba-a11a-c7e21ca61d3f to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)"]}

Comment 18 jcallaha 2019-03-25 20:54:09 UTC

Since the task is indeed being completely in katello (all hypervisors added), and the above failure being a result of the use of a synchronous endpoint, this bug is fixed as a result of https://bugzilla.redhat.com/show_bug.cgi?id=1687190 in Satellite 6.5.0 Snap 21

Comment 20 errata-xmlrpc 2019-05-14 12:40:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222