Bug 1464559

Summary: Deadlock when registering many consumers and consuming subscriptions
Product: Red Hat Satellite Reporter: Chris "Ceiu" Rog <crog>
Component: katello-agentAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.8CC: bbuckingham, bkearney, jcallaha
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-22 19:15:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Archive containing the test script and its required library
none
Manifest containing data used by the test script
none
Stack trace from foreman/production.log none

Description Chris "Ceiu" Rog 2017-06-23 18:24:21 UTC
Created attachment 1291242 [details]
Archive containing the test script and its required library

Description of problem:
While running a stress test to attempt to reproduce a deadlock issue in Candlepin, we managed to reliably trigger a deadlock within Katello.

The stress test creates many consumers, consumes many pools in random order, then unregisters the consumer, all in parallel. At the time of writing, the test runs 50 threads which each run through 50 iterations of creating a consumer, consuming 25 pools (in random order) and then unregistering. This script will be attached to this BZ, as well as the manifest used to set up the test data.

Note that the included manifest will fail signature validation within Candlepin. We worked around this by disabling the check in Candlepin before importing, but another option would be to bypass the check using the force flag(s). If neither of these options are acceptable, any manifest which creates subscriptions with sku products named "export_prod-0" through "export_prod-25" should be acceptable.

Also note that the KatelloModule must be disabled in Candlepin, and simple auth must be enabled to allow direct access to certain endpoints to fetch the proper IDs used by the script.

Version-Release number of selected component (if applicable):
katello-3.0.0-17.el7sat.noarch
tfm-rubygem-katello-3.0.0.105-1.el7sat.noarch

How reproducible:
90%+

Steps to Reproduce:
1. Import manifest containing test data to the default organization
2. Update the IDs in the script to properly match those of the default organization and default environment
3. Run the test script

Actual results:
Deadlock (see attached stack trace)

Expected results:
No deadlock in Katello

Additional info:

Comment 1 Chris "Ceiu" Rog 2017-06-23 18:24:56 UTC
Created attachment 1291243 [details]
Manifest containing data used by the test script

Comment 2 Chris "Ceiu" Rog 2017-06-23 18:40:31 UTC
Created attachment 1291244 [details]
Stack trace from foreman/production.log

Comment 4 Brad Buckingham 2017-09-22 19:15:57 UTC

*** This bug has been marked as a duplicate of bug 1425212 ***