Bug 1317646

Summary: Possible race condition when creating new Products
Product: Red Hat Satellite Reporter: Og Maciel <omaciel>
Component: WebUIAssignee: Christine Fouant <cfouant>
Status: CLOSED ERRATA QA Contact: Stephen Benjamin <stbenjam>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, cduryee, jsherril, stbenjam
Target Milestone: UnspecifiedKeywords: PrioBumpQA, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/14206
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 09:06:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Og Maciel 2016-03-14 18:11:30 UTC
Description of problem:

Our automation has shown a massive number of failures while creating new custom Products. Since the automated jobs are running on at least 4 different threads, we're wondering if this is what uncovered the issue. This could potentially be related to http://projects.theforeman.org/issues/8586 ?

All errors reported show the same pattern:

    2016-03-11 18:54:05 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 941, "name": "\ud849\udf8a"}.
2016-03-11 18:58:23 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}

Version-Release number of selected component (if applicable):

* Satellite 6.2.0 SNAP 3.0

How reproducible:


Steps to Reproduce:
1. I believe you'd have to attempt to create 4+ custom Products at the same time, each with a unique set of values...
2.
3.

Actual results:


Expected results:


Additional info:

More similar errors:

2016-03-11 19:47:14 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 980, "name": "\ud853\udc99\ud868\uddce\ud86c\udff7\ud858\udc3e\ud86b\udd1b\uafbc\ud848\ude01\u422e\ud835\udd7c\ud85c\udfe5\ud854\udf1d\u3e02\ud859\udcf8\ud86c\udfba\ud84f\ude94"}.
2016-03-11 19:53:43 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}


2016-03-11 19:23:18 - nailgun.client - DEBUG - Making HTTP POST request to https://qe-sat6-rhel6.satqe.lab.eng.rdu2.redhat.com/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 963, "name": "\ud866\udc7a\ud859\udf9b\ud86d\ude76\ud863\udd93\ud850\udda5\u85f9\ud86c\ude32\ud855\udc7b\ud84a\udd6c\u9f99\ud85e\ude19"}.
2016-03-11 19:28:25 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}

Comment 1 Justin Sherrill 2016-03-14 18:18:33 UTC
Main causes for this are:

We are indexing ALL subscriptions across the satellite after each product is created.  We really just need to index the subscriptions for this individual product

I would recommend also using Katello::Util::Support.active_record_retry around this to retry the indexing if it fails

Comment 3 Christine Fouant 2016-03-15 17:01:57 UTC
Created redmine issue http://projects.theforeman.org/issues/14206 from this bug

Comment 4 Bryan Kearney 2016-03-15 18:04:44 UTC
Upstream bug component is WebUI

Comment 5 Chris Duryee 2016-03-15 19:22:03 UTC
ideally, we should remove all calls to reindex. barring that, reindexing should not create a race condition when being run 2x at the same time.

Comment 6 Chris Duryee 2016-03-15 19:22:54 UTC
this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1318022. This bug is "indexing should not cause race condition", but other bug is "do not index at all".

Comment 7 Chris Duryee 2016-03-15 19:33:26 UTC
*** Bug 1318022 has been marked as a duplicate of this bug. ***

Comment 8 Bryan Kearney 2016-03-16 16:04:34 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/14206 has been closed
-------------
Christine Fouant
Applied in changeset commit:katello|97dbe26142a31ca4b993b7a6071a35b076b27fff.

Comment 9 Stephen Benjamin 2016-03-24 17:57:24 UTC
Verified:

With a for loop:
  for name in `seq 1 10`; do hammer -u admin -p changeme product create --name "Product $name" --description "Concurrent product creation test" --organization-id 1 & done


  Note the "&" before done - the hammer commands would be run in parallel.

All 10 products are created successfully.

Comment 12 errata-xmlrpc 2016-07-27 09:06:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1500