Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1317646 - Possible race condition when creating new Products
Summary: Possible race condition when creating new Products
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: WebUI
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: Unspecified
Assignee: Christine Fouant
QA Contact: Stephen Benjamin
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1318022 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-14 18:11 UTC by Og Maciel
Modified: 2019-09-26 17:35 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 09:06:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 14206 0 None None None 2016-04-22 15:11:11 UTC
Red Hat Bugzilla 1318022 0 unspecified CLOSED creating a product reindexes all subscriptions, causing slowness 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2016:1500 0 normal SHIPPED_LIVE Red Hat Satellite 6.2 Base Libraries 2016-07-27 12:24:38 UTC

Internal Links: 1318022

Description Og Maciel 2016-03-14 18:11:30 UTC
Description of problem:

Our automation has shown a massive number of failures while creating new custom Products. Since the automated jobs are running on at least 4 different threads, we're wondering if this is what uncovered the issue. This could potentially be related to http://projects.theforeman.org/issues/8586 ?

All errors reported show the same pattern:

    2016-03-11 18:54:05 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 941, "name": "\ud849\udf8a"}.
2016-03-11 18:58:23 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}

Version-Release number of selected component (if applicable):

* Satellite 6.2.0 SNAP 3.0

How reproducible:


Steps to Reproduce:
1. I believe you'd have to attempt to create 4+ custom Products at the same time, each with a unique set of values...
2.
3.

Actual results:


Expected results:


Additional info:

More similar errors:

2016-03-11 19:47:14 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 980, "name": "\ud853\udc99\ud868\uddce\ud86c\udff7\ud858\udc3e\ud86b\udd1b\uafbc\ud848\ude01\u422e\ud835\udd7c\ud85c\udfe5\ud854\udf1d\u3e02\ud859\udcf8\ud86c\udfba\ud84f\ude94"}.
2016-03-11 19:53:43 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}


2016-03-11 19:23:18 - nailgun.client - DEBUG - Making HTTP POST request to https://qe-sat6-rhel6.satqe.lab.eng.rdu2.redhat.com/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 963, "name": "\ud866\udc7a\ud859\udf9b\ud86d\ude76\ud863\udd93\ud850\udda5\u85f9\ud86c\ude32\ud855\udc7b\ud84a\udd6c\u9f99\ud85e\ude19"}.
2016-03-11 19:28:25 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR:  duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}

Comment 1 Justin Sherrill 2016-03-14 18:18:33 UTC
Main causes for this are:

We are indexing ALL subscriptions across the satellite after each product is created.  We really just need to index the subscriptions for this individual product

I would recommend also using Katello::Util::Support.active_record_retry around this to retry the indexing if it fails

Comment 3 Christine Fouant 2016-03-15 17:01:57 UTC
Created redmine issue http://projects.theforeman.org/issues/14206 from this bug

Comment 4 Bryan Kearney 2016-03-15 18:04:44 UTC
Upstream bug component is WebUI

Comment 5 Chris Duryee 2016-03-15 19:22:03 UTC
ideally, we should remove all calls to reindex. barring that, reindexing should not create a race condition when being run 2x at the same time.

Comment 6 Chris Duryee 2016-03-15 19:22:54 UTC
this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1318022. This bug is "indexing should not cause race condition", but other bug is "do not index at all".

Comment 7 Chris Duryee 2016-03-15 19:33:26 UTC
*** Bug 1318022 has been marked as a duplicate of this bug. ***

Comment 8 Bryan Kearney 2016-03-16 16:04:34 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/14206 has been closed
-------------
Christine Fouant
Applied in changeset commit:katello|97dbe26142a31ca4b993b7a6071a35b076b27fff.

Comment 9 Stephen Benjamin 2016-03-24 17:57:24 UTC
Verified:

With a for loop:
  for name in `seq 1 10`; do hammer -u admin -p changeme product create --name "Product $name" --description "Concurrent product creation test" --organization-id 1 & done


  Note the "&" before done - the hammer commands would be run in parallel.

All 10 products are created successfully.

Comment 12 errata-xmlrpc 2016-07-27 09:06:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1500


Note You need to log in before you can comment on or make changes to this bug.