Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1991557 - Many Postgres ERRORs (duplicate key) especially on RedHat repo sync
Summary: Many Postgres ERRORs (duplicate key) especially on RedHat repo sync
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.12.0
Assignee: Justin Sherrill
QA Contact: Cole Higgins
URL:
Whiteboard:
Depends On:
Blocks: 1957813
TreeView+ depends on / blocked
 
Reported: 2021-08-09 12:23 UTC by Brad Buckingham
Modified: 2022-11-16 13:33 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-16 13:32:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 33451 0 Normal Closed Many Postgres ERRORs (duplicate key) especially on RedHat repo sync 2022-02-11 20:09:23 UTC
Red Hat Issue Tracker SAT-7902 0 None None None 2022-09-22 19:04:40 UTC
Red Hat Product Errata RHSA-2022:8506 0 None None None 2022-11-16 13:33:34 UTC

Description Brad Buckingham 2021-08-09 12:23:34 UTC 
Description of problem:

As part of the Satellite 6.9 Pulp 3 High-Touch Beta, several duplicate key errors were observed on Red Hat repository syncs.

During this HTB, a Satellite 6.9 instance went through the process of migration to Pulp 3, a switchover to Pulp 3 was performed and content management workflows executed.

Version-Release number of selected component (if applicable):
6.9.1

Errors observed:

The errors on Thursday are between 03:00am-04:00am that was the first time I was running the RedHat Repositories sync after the pulp3 switchover

To give an idea on the tables and error i summarized the count per hour and error

root@hostname:/var/opt/rh/rh-postgresql12/lib/pgsql/data/log# grep 'UTC ERROR:' *.log | cut -c-1000 | sed 's/\(2021-.* [0-9].:\).*UTC/\1/' | sort | uniq -c
      2 postgresql-Mon.log:2021-06-28 08: ERROR:  duplicate key value violates unique constraint "index_katello_rpms_on_pulp_id"
    843 postgresql-Thu.log:2021-07-01 03: ERROR:  duplicate key value violates unique constraint "index_katello_errata_on_pulp_id"
   2111 postgresql-Thu.log:2021-07-01 03: ERROR:  duplicate key value violates unique constraint "index_katello_erratum_cves_on_erratum_id_and_cve_id_and_href"
   3990 postgresql-Thu.log:2021-07-01 03: ERROR:  duplicate key value violates unique constraint "katello_erratum_bz_eid_bid_href"
  18171 postgresql-Thu.log:2021-07-01 03: ERROR:  duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"problem:

Additional info:

https://bugzilla.redhat.com/show_bug.cgi?id=1991527 - contains additional errors reported

See https://bugzilla.redhat.com/show_bug.cgi?id=1991527#c3
Comment 2 Justin Sherrill 2021-09-01 19:57:33 UTC 
it looks like these errors popped in the logs during sync, this is actually expected when syncing multiple repos at the same time.  Rails now supports upsert and upsert_all (as of rails 6.0) which should allow us to avoid this and possibly improve performance as well.  As long as the actual sync didn't fail (which it doesn't seem like it did), i don't consider this a regression however.  But it should be a small change we can do in 6.10
Comment 3 Justin Sherrill 2021-09-09 20:30:02 UTC 
Created redmine issue https://projects.theforeman.org/issues/33451 from this bug
Comment 4 Bryan Kearney 2021-09-10 00:05:21 UTC 
Upstream bug assigned to jsherril
Comment 5 Bryan Kearney 2021-09-10 00:05:23 UTC 
Upstream bug assigned to jsherril
Comment 6 Justin Sherrill 2021-09-10 11:38:44 UTC 
After some investigation, it turns out some tables are missing uniqueness constraints which are probably too risky to add with migrations at this point in 6.10.  I'm proposing we move to 7.0.   Since this isn't a regression from 6.8/6.9 or earlier I don't see it as a big issue
Comment 7 Peter Vreman 2021-09-14 08:54:07 UTC 
Some input from user point of view:

For me the errors did not impact the functionality.

The minor impact:
- Size the postgres log grows
- Many red herrings in the log that might hide real errors and distract during troubleshooting.


Still it would be good to have this documented as being expected in 6.10 and improvements are on roadmap for 7.x
Comment 11 Bryan Kearney 2022-02-17 16:05:02 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/33451 has been resolved.
Comment 14 Justin Sherrill 2022-04-06 19:56:07 UTC 
I would recommend punting this to 6.12.  The change has already been merged upstream, but it was a very large change and present good bit of risk.  Cherry picking back to 6.11 would not be advisable.  However since its merged against upstream it will automatically appear in 6.12, and have more testing/baking done.
Comment 19 errata-xmlrpc 2022-11-16 13:32:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506
Note You need to log in before you can comment on or make changes to this bug.