Bug 2030714

Summary:	Upgrade to Satellite 6.10 fails on content-switchover - Postgresql: duplicate key value violates unique constraint "index_katello_rpms_on_pulp_id"
Product:	Red Hat Satellite	Reporter:	Jaroslav Krajicek <jkrajice>
Component:	Repositories	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED ERRATA	QA Contact:	Akhil Jha <akjha>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.9.7	CC:	ahumbe, akjha, jpasqual, jsherril, pdudley, pmendezh, pmoravec, sadas, saydas, wpinheir, zhunting
Target Milestone:	6.9.8	Keywords:	PrioBumpGSS, Triaged, Upgrades
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-3.18.1.49-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-01-27 17:33:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jaroslav Krajicek 2021-12-09 14:53:43 UTC

Description of problem:

Upgrade to Satellite 6.10 fails on katello:pulp3_content_switchover, blocking further steps.

~~~
Failed executing foreman-rake katello:pulp3_content_switchover, exit status 1:
 API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_rpms_on_pulp_id"
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/rpm/packages/979d5477-4c2e-4994-9367-9ef5d3387258/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
~~~

How reproducible:
 Reverting and re-run results in the same

Additional info:
 Attached trace output

Comment 2 Justin Sherrill 2021-12-09 15:04:47 UTC

do we have a reproducer?  or would it be possible to get the foreman & pulpcore postgresql databases from a customer?

Comment 12 Paul Dudley 2021-12-11 00:31:26 UTC

Attached another customer case to this, errors seem to fit the pattern:
~~~
Several variant IDs of this:
374:2021-12-10 13:07:56 EST ERROR:  duplicate key value violates unique constraint "core_artifact_sha256_key"
375-2021-12-10 13:07:56 EST DETAIL:  Key (sha256)=(d0cbb960d9e7579b29c3a946c344b7efa4f4bab6e357f9f2c1a9ced8ce0ff518) already exists.
--
A couple copies of these, one for each content-switchover attempt after the issue:
377:2021-12-10 13:25:41 EST ERROR:  duplicate key value violates unique constraint "index_katello_rpms_on_pulp_id"
378-2021-12-10 13:25:41 EST DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/rpm/packages/6f0e4a46-3556-4c26-96cb-1fad61475c0d/) already exists.
~~~

No errors observed in Migration tasks. Initial upgrade attempt failed due to yum dependency issue, not related to migrating content. Confirmed that in between the pre-migration tasks and the resolution of the dependency issue (thus another run of the content-switchover) two repos were synced. Unable to confirm exactly these things as we don't have the DB, but the case seems to fit scenario in comment 8 exactly.

Comment 14 Justin Sherrill 2021-12-13 18:00:28 UTC

Plan of action:

1) create a rake task that will attempt to cleanup data from pulp2's usage after switchover
2) Add checks on 6.9.z at sync time, cv publish/promote, package upload to stop any actions if switchover has been performed
3) update foreman-maintain/satellite-maintain to warn users that once switchover completes, they cannot go back to 6.9

Comment 17 Justin Sherrill 2021-12-16 18:12:22 UTC

Connecting redmine issue https://projects.theforeman.org/issues/34155 from this bug

Comment 19 Bryan Kearney 2022-01-06 00:05:12 UTC

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34155 has been resolved.

Comment 24 Akhil Jha 2022-01-25 11:03:15 UTC

Verified sat 6.9.8-snap2.0

Steps followed:
1) Synced repos and created few custom products,LCE,CV
2) Upgrade to 6.10 successful (Steps: https://access.redhat.com/documentation/en-us/red_hat_satellite/6.10/html-single/upgrading_and_updating_red_hat_satellite/index#preparing_to_migrate_pulp_content)

Comment 30 errata-xmlrpc 2022-01-27 17:33:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.8 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0320