Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Our automation has shown a massive number of failures while creating new custom Products. Since the automated jobs are running on at least 4 different threads, we're wondering if this is what uncovered the issue. This could potentially be related to http://projects.theforeman.org/issues/8586 ?
All errors reported show the same pattern:
2016-03-11 18:54:05 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 941, "name": "\ud849\udf8a"}.
2016-03-11 18:58:23 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 3bd1b5be-5379-4d6f-98fb-4de4b40cb26c: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}
Version-Release number of selected component (if applicable):
* Satellite 6.2.0 SNAP 3.0
How reproducible:
Steps to Reproduce:
1. I believe you'd have to attempt to create 4+ custom Products at the same time, each with a unique set of values...
2.
3.
Actual results:
Expected results:
Additional info:
More similar errors:
2016-03-11 19:47:14 - nailgun.client - DEBUG - Making HTTP POST request to https://<EDITED>/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 980, "name": "\ud853\udc99\ud868\uddce\ud86c\udff7\ud858\udc3e\ud86b\udd1b\uafbc\ud848\ude01\u422e\ud835\udd7c\ud85c\udfe5\ud854\udf1d\u3e02\ud859\udcf8\ud86c\udfba\ud84f\ude94"}.
2016-03-11 19:53:43 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task 721c84c1-25e8-4fa2-9d72-38e52faffa39: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}
2016-03-11 19:23:18 - nailgun.client - DEBUG - Making HTTP POST request to https://qe-sat6-rhel6.satqe.lab.eng.rdu2.redhat.com/katello/api/v2/products with options {'verify': False, 'auth': ('admin', 'changeme'), 'headers': {'content-type': 'application/json'}} and data {"organization_id": 963, "name": "\ud866\udc7a\ud859\udf9b\ud86d\ude76\ud863\udd93\ud850\udda5\u85f9\ud86c\ude32\ud855\udc7b\ud84a\udd6c\u9f99\ud85e\ude19"}.
2016-03-11 19:28:25 - nailgun.client - WARNING - Received HTTP 500 response: {"displayMessage":"Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\"","errors":["Task a25383a4-a305-45f3-8526-ec485aebf78d: ActiveRecord::RecordNotUnique: PG::Error: ERROR: duplicate key value violates unique constraint \"index_katello_subscriptions_on_cp_id\"\n: INSERT INTO \"katello_subscriptions\" (\"cp_id\", \"created_at\", \"updated_at\") VALUES ($1, $2, $3) RETURNING \"id\""]}
Main causes for this are:
We are indexing ALL subscriptions across the satellite after each product is created. We really just need to index the subscriptions for this individual product
I would recommend also using Katello::Util::Support.active_record_retry around this to retry the indexing if it fails
Moving to POST since upstream bug http://projects.theforeman.org/issues/14206 has been closed
-------------
Christine Fouant
Applied in changeset commit:katello|97dbe26142a31ca4b993b7a6071a35b076b27fff.
Verified:
With a for loop:
for name in `seq 1 10`; do hammer -u admin -p changeme product create --name "Product $name" --description "Concurrent product creation test" --organization-id 1 & done
Note the "&" before done - the hammer commands would be run in parallel.
All 10 products are created successfully.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2016:1500