Bug 2075519

Summary: Upgrade fails during db:migrate with PG::ForeignKeyViolation: ERROR: update or delete on table "katello_errata" violates foreign key constraint "katello_content_facet_errata_errata_id"
Product: Red Hat Satellite Reporter: Lukas Pramuk <lpramuk>
Component: Errata ManagementAssignee: Samir Jha <sajha>
Status: CLOSED ERRATA QA Contact: Akhil Jha <akjha>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.11.0CC: ehelms, jbhatia, sajha, swadeley
Target Milestone: 6.11.0Keywords: Triaged, UpgradeBlocker, Upgrades
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-katello-4.3.0.23 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2077046 (view as bug list) Environment:
Last Closed: 2022-07-05 14:35:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2077046    

Description Lukas Pramuk 2022-04-14 12:17:48 UTC 
Description of problem:
Satellite upgrade fails during db:migrate with PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_errata" violates foreign key constraint "katello_content_facet_errata_errata_id" on table "katello_content_facet_errata"

QE populated Satellite was used.

Version-Release number of selected component (if applicable):
6.11.0 Snap16

How reproducible:
deterministic

Steps to Reproduce:
1. Have a QE populated and registered Satellite 6.10 
2. Upgrade to 6.11.0

# satellite-maintain upgrade run --target-version 6.11 -w repositories-validate,repositories-setup -y
...

Running Migration scripts to Satellite 6.11
================================================================================
Setup repositories:                                                   [SKIPPED]
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 6:
 2022-04-14 07:12:06 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-04-14 07:12:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-04-14 07:12:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-04-14 07:12:20 [WARN  ] [pre] Skipping system checks.
2022-04-14 07:12:20 [WARN  ] [pre] Skipping system checks.
2022-04-14 07:12:33 [NOTICE] [configure] Starting system configuration.
2022-04-14 07:12:49 [NOTICE] [configure] 250 configuration steps out of 2165 steps complete.
2022-04-14 07:13:02 [NOTICE] [configure] 500 configuration steps out of 2169 steps complete.
2022-04-14 07:13:12 [NOTICE] [configure] 750 configuration steps out of 2171 steps complete.
2022-04-14 07:13:23 [NOTICE] [configure] 1000 configuration steps out of 2176 steps complete.
2022-04-14 07:13:38 [NOTICE] [configure] 1250 configuration steps out of 2194 steps complete.
2022-04-14 07:14:58 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-14 07:14:58 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-14 07:16:19 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-14 07:16:19 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-04-14 07:16:49 [NOTICE] [configure] 1500 configuration steps out of 2195 steps complete.
2022-04-14 07:16:53 [NOTICE] [configure] 1750 configuration steps out of 3039 steps complete.
2022-04-14 07:16:54 [NOTICE] [configure] 2000 configuration steps out of 3039 steps complete.
2022-04-14 07:16:54 [NOTICE] [configure] 2250 configuration steps out of 3039 steps complete.
2022-04-14 07:16:55 [NOTICE] [configure] 2500 configuration steps out of 3039 steps complete.
2022-04-14 07:17:15 [NOTICE] [configure] 2750 configuration steps out of 3039 steps complete.
2022-04-14 07:17:52 [NOTICE] [configure] 3000 configuration steps out of 3039 steps complete.
2022-04-14 07:18:10 [NOTICE] [configure] System configuration has finished.
...

Actual results:
failed upgrade due to failing db:migrate

Expected results:
successful upgrade
Comment 1 Lukas Pramuk 2022-04-14 12:31:23 UTC 
# foreman-rake db:migrate
'ErbParser' is ignored.
'RubyParser' is ignored.
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
== 20220303160220 RemoveDuplicateErrata: migrating ============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_errata" violates foreign key constraint "katello_content_facet_errata_errata_id" on table "katello_content_facet_errata"
DETAIL:  Key (id)=(12675) is still referenced from table "katello_content_facet_errata".
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:656:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:111:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:180:in `delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:587:in `delete_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.20/db/migrate/20220303160220_remove_duplicate_errata.rb:32:in `block in up'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.20/db/migrate/20220303160220_remove_duplicate_errata.rb:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.20/db/migrate/20220303160220_remove_duplicate_errata.rb:16:in `up'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:831:in `exec_migration'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:811:in `block in migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:810:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1002:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1361:in `block in ddl_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1361:in `ddl_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1280:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1229:in `block in migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1382:in `with_advisory_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1229:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1061:in `up'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/migration.rb:1036:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/databases.rake:84:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
...
Comment 3 Bryan Kearney 2022-04-20 20:05:10 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34771 has been resolved.
Comment 4 Akhil Jha 2022-05-05 17:47:15 UTC 
Verified.
Upgraded to 6.11.0-19.0

# rpm -qa | grep rubygem-katello
tfm-rubygem-katello-4.3.0.33-1.el7sat.noarch

Upgrade successful.

Thanks sajha for helping in verifying the BZ.
Comment 7 errata-xmlrpc 2022-07-05 14:35:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498