Bug 1404436

Summary: Satellite 6.1.X -> 6.2.5 upgrade fails during db:migration
Product: Red Hat Satellite Reporter: Mike McCune <mmccune>
Component: UpgradesAssignee: Tomer Brisker <tbrisker>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.2.5CC: ahumbe, chrobert, dmoessne, egolov, ehelms, h.teunis, inecas, jcallaha, jko, mbacovsk, omaciel, oshtaier, sghai, tbrisker, tcarlin
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-1.11.0.61-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-19 08:18:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
hotfix patch for Foreman
none
logs from console after applying patch and while running upgrade
none
good upgrade 1
none
full.log none

Description Mike McCune 2016-12-13 20:50:39 UTC
1) Upgrade from Satellite 6.2.4 to 6.2.5 and receive this error:


** Execute db:load_config
** Execute db:migrate
== 20151104100257 AddHostsCountToHostgroup: migrating =========================
-- add_column(:hostgroups, :hosts_count, :integer, {:default=>0})
   -> 0.0454s
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

undefined method `counter_cache_column' for nil:NilClass/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/counter_cache.rb:33:in `block in reset_counters'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/counter_cache.rb:22:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/counter_cache.rb:22:in `reset_counters'
/usr/share/foreman/db/migrate/20151104100257_add_hosts_count_to_hostgroup.rb:5:in `block in up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/usr/share/foreman/db/migrate/20151104100257_add_hosts_count_to_hostgroup.rb:4:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'
/opt/rh/rh-ruby22/root/usr/share/ruby/benchmark.rb:288:in `measure'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in `migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in `migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in `block in execute_migration_in_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `block in ddl_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `ddl_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in `execute_migration_in_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in `block in migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in `migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'

Comment 2 Mike McCune 2016-12-13 21:15:40 UTC
Created redmine issue http://projects.theforeman.org/issues/17666 from this bug

Comment 3 Mike McCune 2016-12-13 21:47:56 UTC
This appears to only effect 6.1 -> 6.2.5 upgrades.

Going from 6.2.X -> 6.2.5 still functions fine.

Comment 4 Tomer Brisker 2016-12-13 21:49:51 UTC
This also requires at least one hostgroup in the database before attempting 6.1->6.2.5 upgrade.

Comment 6 Mike McCune 2016-12-13 22:17:43 UTC
*** HOTFIX PATCH ***

For anyone wishing to work around this and complete an upgrade to 6.2.5 please perform the following:

1) Download attached patch from this BZ

2) copy to:

/usr/share/foreman

3) Apply patch;

# patch -p1 < 4109.patch 
patching file db/migrate/20140314004243_add_counter_caches.rb
patching file db/migrate/20140318153157_fix_puppetclass_counters.rb
patching file db/migrate/20151104100257_add_hosts_count_to_hostgroup.rb

4) continue your upgrade:

satellite-installer --upgrade

Comment 7 Mike McCune 2016-12-13 22:18:12 UTC
Created attachment 1231344 [details]
hotfix patch for Foreman

Comment 8 Bryan Kearney 2016-12-13 23:20:12 UTC
Upstream bug assigned to tbrisker

Comment 9 Bryan Kearney 2016-12-13 23:20:16 UTC
Upstream bug assigned to tbrisker

Comment 10 Sachin Ghai 2016-12-14 08:10:01 UTC
Issue is reproducible with 6.1.11 -> 6.2.5 and I created a hostgroup before upgrade.


----
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
true

rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

undefined method `counter_cache_column' for nil:NilClass/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/counter_cache.rb:33:in `block in reset_counters'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/counter_cache.rb:22:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/counter_cache.rb:22:in `reset_counters'
/usr/share/foreman/db/migrate/20151104100257_add_hosts_count_to_hostgroup.rb:5:in `block in up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/usr/share/foreman/db/migrate/20151104100257_add_hosts_count_to_hostgroup.rb:4:in `up'

Comment 13 Sachin Ghai 2016-12-14 11:11:22 UTC
Issue pointed in comment11 appears because I was trying to run installer from /usr/share/foreman. However, it ran successfully when I run it from `cd $HOME`.

filed this bz to address this:https://bugzilla.redhat.com/show_bug.cgi?id=1404652

Comment 14 Sachin Ghai 2016-12-14 11:13:13 UTC
Applied patch from comment6 and ran "satellite-installer --scenario satellite --upgrade" from /root and upgrade was completed successfully. Please see the complete console logs.

Comment 15 Sachin Ghai 2016-12-14 11:14:14 UTC
Created attachment 1231607 [details]
logs from console after applying patch and while running upgrade

Comment 16 jcallaha 2016-12-14 22:00:00 UTC
Created attachment 1231909 [details]
good upgrade 1

attaching log from a good upgrade. satellite did have hostgroups.

Comment 17 jcallaha 2016-12-16 05:23:24 UTC
Created attachment 1232434 [details]
full.log

Verified in Satellite 6.2.6 Snap 2

Completed a final upgrade with a fully populated satellite, to include multiple configured hostgroups, and hosts provisioned from those groups. Attaching the upgrade log to this update.

Comment 19 errata-xmlrpc 2016-12-19 08:18:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2958