Bug 1853200

Summary:	foreman-rake-db:migrate Fails on "No indexes found on foreman_tasks_locks with the options provided"
Product:	Red Hat Satellite	Reporter:	Devendra Singh <desingh>
Component:	Tasks Plugin	Assignee:	Adam Ruzicka <aruzicka>
Status:	CLOSED ERRATA	QA Contact:	Peter Ondrejka <pondrejk>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.8.0	CC:	aruzicka, jsherril, pcreech, tbrisker
Target Milestone:	6.8.0	Keywords:	AutomationBlocker, UpgradeBlocker, Upgrades
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	tfm-rubygem-foreman-tasks-2.0.2, tfm-rubygem-foreman-tasks-1.1.3	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 13:03:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Devendra Singh 2020-07-02 07:04:38 UTC

Description of problem: foreman-rake-db:migrate command return 1 instead of 0 while upgrading the satellite from 6.7 to 6.8.

Version-Release number of selected component (if applicable):
6.8 Snap7

How reproducible:
always

Steps to Reproduce:
1. Prepare the environment to perform the satellite-upgrade from 6.7 to 6.8 
2. Configure the all required repository to perform the upgrade from 6.7 to 6.8.
3. Apply the workaround mentioned on BZ#1846593#c5
4: Run foreman-maintain upgrade check and it completes with some expected warning messages.

5: Run foreman upgrade using below command

foreman-maintain upgrade run --whitelist="disk-performance, repositories-validate, repositories-setup" --target-version 6.8 -y

Running Migration scripts to Satellite 6.8

 ===========================================================================
 Setup repositories:                                                   [SKIPPED]
 ------------------------------------------------------------------------------
 Unlock packages:                                                      [OK]
 ------------------------------------------------------------------------------

 Update package(s) :                                                   [OK]
 ------------------------------------------------------------------------------

 Procedures::Installer::Upgrade:                                       [FAIL]
 Failed executing LANG=en_US.utf-8 satellite-installer, exit status 6:

Hook /usr/share/foreman-installer/katello/hooks/post/99-version_locking.rb returned true
All hooks in group post finished
Exit with status code: 6 (signal was 6)
Errors encountered during run:
'/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:183:in `sync'
 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:241:in `sync'
 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:136:in `sync_if_needed'
 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:90:in `block in perform_changes'
 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `each'
 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `perform_changes'
 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'


Actual results:
foreman-rake-db:migrate return 1 instead of 0

Expected results:
foreman-rake-db:migrate should be completed successfully.


Additional info:

Comment 2 Justin Sherrill 2020-07-02 12:48:33 UTC

relevant error:

No indexes found on foreman_tasks_locks with the options provided.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:1293:in `index_name_for_remove'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:462:in `remove_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:890:in `block in method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:858:in `block in say_with_time'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:858:in `say_with_time'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:879:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.1/db/migrate/20200611090846_add_task_lock_index_on_resource_type_and_task_id.rb:7:in `change'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:828:in `exec_migration'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:811:in `block in migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:810:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1002:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1361:in `block in ddl_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1361:in `ddl_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1280:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1229:in `block in migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1382:in `with_advisory_lock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1229:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1061:in `up'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/migration.rb:1036:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/databases.rake:84:in `each'

Comment 3 Adam Ruzicka 2020-07-02 13:26:10 UTC

Created redmine issue https://projects.theforeman.org/issues/30311 from this bug

Comment 4 Bryan Kearney 2020-07-04 20:03:26 UTC

Upstream bug assigned to aruzicka

Comment 5 Bryan Kearney 2020-07-04 20:03:27 UTC

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/30311 has been resolved.

Comment 7 Peter Ondrejka 2020-07-21 12:09:44 UTC

Verified on Satellite 6.8 snap 9, via upgrade automation, the aforementioned error no longer occurs

Comment 8 Devendra Singh 2020-07-21 12:30:11 UTC

Verification points:

* Package version verified.

  #rpm -q tfm-rubygem-foreman-tasks-2.0.2
  tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7sat.noarch

* satellite-installer successfully completed in upgrade.

foreman-maintain upgrade run --whitelist="disk-performance, repositories-validate, repositories-setup" --target-version 6.8 -y
.......
.......
 ================================================================================
 Setup repositories:                                                   [SKIPPED]
 --------------------------------------------------------------------------------
 Unlock packages:                                                      [OK]
 --------------------------------------------------------------------------------
 Update package(s) :                                                   [OK]
 --------------------------------------------------------------------------------
 Procedures::Installer::Upgrade:                                       [OK]
 --------------------------------------------------------------------------------
 
 
 Running Procedures after migrating to Satellite 6.8
 ==========================================================
 | All services started                                                [OK]
 --------------------------------------------------------------------------------
 re-enable sync plans: 
 / Total 3 sync plans are now enabled.                                 [OK]
 --------------------------------------------------------------------------------
 Remove maintenance_mode chain from iptables:                          [OK]
 --------------------------------------------------------------------------------
 
 
 Running Checks after upgrading to Satellite 6.8
 ================================================================================
 Clean old Kernel and initramfs files from tftp-boot:                  [OK]
 --------------------------------------------------------------------------------
 Check number of fact names in database:                               [OK]
 --------------------------------------------------------------------------------
 Check for verifying syntax for ISP DHCP configurations:               [OK]
 --------------------------------------------------------------------------------
 Check whether all services are running:                               [OK]
 --------------------------------------------------------------------------------
 Check whether all services are running using the ping call:           [OK]
 --------------------------------------------------------------------------------
 Check for paused tasks:                                               [OK]
 --------------------------------------------------------------------------------
 Check to verify no empty CA cert requests exist:                      [OK]
 --------------------------------------------------------------------------------
 Check whether system is self-registered or not:                       [OK]
 --------------------------------------------------------------------------------

Comment 11 errata-xmlrpc 2020-10-27 13:03:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366