Bug 1514508 - [RFE] allow registrations to occur without blocking on task completion
Summary: [RFE] allow registrations to occur without blocking on task completion
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Registration
Version: 6.2.12
Hardware: Unspecified
OS: Unspecified
high vote
Target Milestone: Unspecified
Assignee: Chris Duryee
QA Contact: jcallaha
: 1438945 1523254 (view as bug list)
Depends On:
Blocks: 1353215 1122832 1490019
TreeView+ depends on / blocked
Reported: 2017-11-17 15:52 UTC by Chris Duryee
Modified: 2021-12-10 15:25 UTC (History)
24 users (show)

Fixed In Version: rubygem-katello-
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1553861 1553862 1553869 1553871 (view as bug list)
Last Closed: 2018-05-21 20:16:43 UTC
Target Upstream Version:

Attachments (Terms of Use)
verification gif (81.75 KB, image/gif)
2018-04-27 19:04 UTC, jcallaha
no flags Details

System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 21703 0 Normal Closed allow system registrations to happen without waiting on tasks to complete 2020-11-26 09:49:03 UTC
Foreman Issue Tracker 21797 0 Normal Closed do not log stack trace if generateapplicability generates a 404 2020-11-26 09:49:04 UTC
Red Hat Product Errata RHBA-2018:1672 0 None None None 2018-05-21 20:17:45 UTC

Description Chris Duryee 2017-11-17 15:52:42 UTC
Description of problem:

Currently, the system registration and unregistration processes create tasks, and then wait for those tasks to complete before returning to subscription-manager.

This usually works fine, but if the satellite is heavily loaded, registration requests may hold passenger workers for a long period of time while tasks are processed. This can eventually cause the satellite to run out of passenger workers and will tip over. Additionally, if http requests get backed up and take all queue slots, pulp API requests will become unresponsive. Many tasks (including system registration) require Pulp to accept API requests, so this can result in the satellite getting wound around the axle.

If registrations and unregistrations are done without blocking on tasks, this avoids this issue. It's technically not an issue if unregistrations occur via task, but if a register happens without a task, we would want the unregister to happen in the same time frame to ensure the subscription is freed before its consumed elsewhere.

Version-Release number of selected component (if applicable): 6.2.12

How reproducible: every time

Steps to Reproduce:

** note ** these steps describe how to make the satellite slow but not fall over from registrations, so the issue is more apparent. Once it falls over, there is not much to observe :)

1. set up a new satellite and configure it to have 40 passenger workers. The number isn't important as long as it's higher than the number in step 2
2. set up 15 clients to register to the Satellite in a loop, using subscrpition-mangaer register --force
3. on the satellite, run 'watch "passenger-status --show=requests | grep -e 'uri\|connected'" '. This will show the response times for the registration requests as they come in.

Actual results: lots of registration requests over 10 seconds, some over 20-30 seconds. Lots of running/pending tasks in the task list.

Expected results: registrations should get processed within a few seconds so as not to hold up passenger workers

Comment 3 Mike McCune 2017-12-11 16:34:06 UTC
*** Bug 1438945 has been marked as a duplicate of this bug. ***

Comment 10 Satellite Program 2018-02-15 13:15:38 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21703 has been resolved.

Comment 16 Brad Buckingham 2018-03-05 15:57:20 UTC
*** Bug 1523254 has been marked as a duplicate of this bug. ***

Comment 23 jcallaha 2018-04-27 19:04:24 UTC
Verified in Satellite 6.2.15 Snap 2

I loosely followed the steps outlined in the description.

I increased the passengers to 40
Then ran a loop of 150 hosts creating, registering, and destroying themselves in batches of 15.

See the attached gif for a recording of the grep'd passenger-status.

Comment 24 jcallaha 2018-04-27 19:04:59 UTC
Created attachment 1427796 [details]
verification gif

Comment 27 errata-xmlrpc 2018-05-21 20:16:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.