Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1317646 - Possible race condition when creating new Products
Possible race condition when creating new Products
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: WebUI (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity urgent (vote)
: Beta
: Unused
Assigned To: Christine Fouant
Stephen Benjamin
http://projects.theforeman.org/issues...
: PrioBumpQA, Triaged
: 1318022 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-14 14:11 EDT by Og Maciel
Modified: 2016-07-27 05:06 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 05:06:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 14206 None None None 2016-04-22 11:11 EDT
Red Hat Product Errata RHBA-2016:1500 normal SHIPPED_LIVE Red Hat Satellite 6.2 Base Libraries 2016-07-27 08:24:38 EDT

  None (edit)
Description Og Maciel 2016-03-14 14:11:30 EDT
Description of problem:

Our automation has shown a massive number of failures while creating new custom Products. Since the automated jobs are running on at least 4 different threads, we're wondering if this is what uncovered the issue. This could potentially be related to http://projects.theforeman.org/issues/8586 ?

All errors reported show the same pattern:

    2016-03-11 18:54:05 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 941, "name": "\ud849\udf8a"}.
2016-03-11 18:58:23 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}

Version-Release number of selected component (if applicable):

* Satellite 6.2.0 SNAP 3.0

How reproducible:


Steps to Reproduce:
1. I believe you'd have to attempt to create 4+ custom Products at the same time, each with a unique set of values...
2.
3.

Actual results:


Expected results:


Additional info:

More similar errors:

2016-03-11 19:47:14 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 980, "name": "\ud853\udc99\ud868\uddce\ud86c\udff7\ud858\udc3e\ud86b\udd1b\uafbc\ud848\ude01\u422e\ud835\udd7c\ud85c\udfe5\ud854\udf1d\u3e02\ud859\udcf8\ud86c\udfba\ud84f\ude94"}.
2016-03-11 19:53:43 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}


2016-03-11 19:23:18 - nailgun.client - DEBUG - Making HTTP POST request to https://qe-sat6-rhel6.satqe.lab.eng.rdu2.redhat.com/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 963, "name": "\ud866\udc7a\ud859\udf9b\ud86d\ude76\ud863\udd93\ud850\udda5\u85f9\ud86c\ude32\ud855\udc7b\ud84a\udd6c\u9f99\ud85e\ude19"}.
2016-03-11 19:28:25 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}
Comment 1 Justin Sherrill 2016-03-14 14:18:33 EDT
Main causes for this are:

We are indexing ALL subscriptions across the satellite after each product is created.  We really just need to index the subscriptions for this individual product

I would recommend also using Katello::Util::Support.active_record_retry around this to retry the indexing if it fails
Comment 3 Christine Fouant 2016-03-15 13:01:57 EDT
Created redmine issue http://projects.theforeman.org/issues/14206 from this bug
Comment 4 Bryan Kearney 2016-03-15 14:04:44 EDT
Upstream bug component is WebUI
Comment 5 Chris Duryee 2016-03-15 15:22:03 EDT
ideally, we should remove all calls to reindex. barring that, reindexing should not create a race condition when being run 2x at the same time.
Comment 6 Chris Duryee 2016-03-15 15:22:54 EDT
this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1318022. This bug is "indexing should not cause race condition", but other bug is "do not index at all".
Comment 7 Chris Duryee 2016-03-15 15:33:26 EDT
*** Bug 1318022 has been marked as a duplicate of this bug. ***
Comment 8 Bryan Kearney 2016-03-16 12:04:34 EDT
Moving to POST since upstream bug http://projects.theforeman.org/issues/14206 has been closed
-------------
Christine Fouant
Applied in changeset commit:katello|97dbe26142a31ca4b993b7a6071a35b076b27fff.
Comment 9 Stephen Benjamin 2016-03-24 13:57:24 EDT
Verified:

With a for loop:
  for name in `seq 1 10`; do hammer -u admin -p changeme product create --name "Product $name" --description "Concurrent product creation test" --organization-id 1 & done


  Note the "&" before done - the hammer commands would be run in parallel.

All 10 products are created successfully.
Comment 12 errata-xmlrpc 2016-07-27 05:06:29 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1500

Note You need to log in before you can comment on or make changes to this bug.