Description of problem: Our automation has shown a massive number of failures while creating new custom Products. Since the automated jobs are running on at least 4 different threads, we're wondering if this is what uncovered the issue. This could potentially be related to http://projects.theforeman.org/issues/8586 ? All errors reported show the same pattern: 2016-03-11 18:54:05 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 941, "name": "\ud849\udf8a"}. 2016-03-11 18:58:23 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]} Version-Release number of selected component (if applicable): * Satellite 6.2.0 SNAP 3.0 How reproducible: Steps to Reproduce: 1. I believe you'd have to attempt to create 4+ custom Products at the same time, each with a unique set of values... 2. 3. Actual results: Expected results: Additional info: More similar errors: 2016-03-11 19:47:14 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 980, "name": "\ud853\udc99\ud868\uddce\ud86c\udff7\ud858\udc3e\ud86b\udd1b\uafbc\ud848\ude01\u422e\ud835\udd7c\ud85c\udfe5\ud854\udf1d\u3e02\ud859\udcf8\ud86c\udfba\ud84f\ude94"}. 2016-03-11 19:53:43 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]} 2016-03-11 19:23:18 - nailgun.client - DEBUG - Making HTTP POST request to https://qe-sat6-rhel6.satqe.lab.eng.rdu2.redhat.com/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 963, "name": "\ud866\udc7a\ud859\udf9b\ud86d\ude76\ud863\udd93\ud850\udda5\u85f9\ud86c\ude32\ud855\udc7b\ud84a\udd6c\u9f99\ud85e\ude19"}. 2016-03-11 19:28:25 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}
Main causes for this are: We are indexing ALL subscriptions across the satellite after each product is created. We really just need to index the subscriptions for this individual product I would recommend also using Katello::Util::Support.active_record_retry around this to retry the indexing if it fails
Created redmine issue http://projects.theforeman.org/issues/14206 from this bug
Upstream bug component is WebUI
ideally, we should remove all calls to reindex. barring that, reindexing should not create a race condition when being run 2x at the same time.
this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1318022. This bug is "indexing should not cause race condition", but other bug is "do not index at all".
*** Bug 1318022 has been marked as a duplicate of this bug. ***
Moving to POST since upstream bug http://projects.theforeman.org/issues/14206 has been closed ------------- Christine Fouant Applied in changeset commit:katello|97dbe26142a31ca4b993b7a6071a35b076b27fff.
Verified: With a for loop: for name in `seq 1 10`; do hammer -u admin -p changeme product create --name "Product $name" --description "Concurrent product creation test" --organization-id 1 & done Note the "&" before done - the hammer commands would be run in parallel. All 10 products are created successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1500