Bug 1688840 - Candlepin connection times out on large virt-who checkins
Summary: Candlepin connection times out on large virt-who checkins
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Candlepin
Version: 6.5.0
Hardware: Unspecified
OS: Unspecified
high vote
Target Milestone: 6.5.0
Assignee: Barnaby Court
QA Contact: jcallaha
Depends On:
TreeView+ depends on / blocked
Reported: 2019-03-14 14:59 UTC by jcallaha
Modified: 2019-11-05 22:47 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-05-14 12:40:29 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:40:36 UTC

Description jcallaha 2019-03-14 14:59:21 UTC
Description of problem:
When submitting a large number of hypervisors to the candlepin event, either via virt-who or manually, the Hypervisors task will almost always fail due to:

Katello::Resources::Candlepin::Consumer: Timed out connecting to server  (GET /candlepin/consumers/b709bdb4-bfc9-4ad8-9841-9f1c49fb77cd)

Version-Release number of selected component (if applicable):
Satellite 6.5.0 Snap 19

How reproducible:
Almost always

Steps to Reproduce:
1. docker run --rm -e "SATHOST=$(hostname)" -e "HYPERVISORS=5000" -e "GUESTS=1" jacobcallahan/genvirt

Actual results:
The task times out with the error above

Expected results:
The tasks process all data and completes

Comment 16 Justin Sherrill 2019-03-25 16:54:52 UTC
This sounds a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1687190

can you reproduce with the newest snap (21 or newer)?

Comment 17 jcallaha 2019-03-25 20:27:52 UTC
Justin, with the upgrade to snap 21, I'm getting something interesting.

The task on in Katello now passes at 5k hypervisors (and in about an hour), but I get this error message from the initial data submittion:

{"displayMessage":"The time waiting for task 0fc23af2-a13d-42ba-a11a-c7e21ca61d3f to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)","errors":["The time waiting for task 0fc23af2-a13d-42ba-a11a-c7e21ca61d3f to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)"]}

Comment 18 jcallaha 2019-03-25 20:54:09 UTC
Since the task is indeed being completely in katello (all hypervisors added), and the above failure being a result of the use of a synchronous endpoint, this bug is fixed as a result of https://bugzilla.redhat.com/show_bug.cgi?id=1687190 in Satellite 6.5.0 Snap 21

Comment 20 errata-xmlrpc 2019-05-14 12:40:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.