Bug 1338737 - Upgrade step db_migrate failed on RHEL6
Summary: Upgrade step db_migrate failed on RHEL6
Keywords:
Status: CLOSED DUPLICATE of bug 1336941
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks: 1335807
TreeView+ depends on / blocked
 
Reported: 2016-05-23 11:07 UTC by Lukas Pramuk
Modified: 2016-05-26 07:47 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-24 13:12:01 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Lukas Pramuk 2016-05-23 11:07:07 UTC
Description of problem:
Upgrade step db_migrate failed on RHEL6


Version-Release number of selected component (if applicable):
Sat6.2.0-Snap12

How reproducible:
always

Steps to Reproduce:
# satellite-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Could not evaluate: Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Failed to call refresh: Proxy <SAT_FQDN> cannot be registered (Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A): N/A
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Proxy <SAT_FQDN> cannot be registered (Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A): N/A
Installing             Done                                               [100%] [....................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
/usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `error': wrong number of arguments (49 for 1) (ArgumentError)
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `send'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:98:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:103:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:28:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:58:in `fail_and_exit'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:52:in `upgrade_step'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:65:in `load'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:51:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:412:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `call'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:411:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:149:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:156:in `run'
	from /usr/sbin/foreman-installer:12

Actual results:
db_migrate failed due to not running services

Expected results:
db_migrate succeeds

Additional info:
After the db_seed failed due to failed db_migrate, I was able to recover by just reruning db_migrate (services were started before db_seed)

Comment 1 Lukas Pramuk 2016-05-23 11:09:04 UTC
/var/log/foreman/db_migrate.log


== 20150817102538 AddDelayAttributes: migrating ===============================
-- add_column(:foreman_tasks_tasks, :start_at, :datetime, {:index=>true, :default=>nil, :null=>true})
   -> 0.0009s
-- add_column(:foreman_tasks_tasks, :start_before, :datetime, {:index=>true, :default=>nil, :null=>true})rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"400 Bad Request"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 400 Bad Request  (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'


== 20150930183738 MigrateContentHosts: migrating ==============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/6.0.24</h3></body></html> (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"4"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'


== 20150114210634 RenameScaptimonyArfReportRawRaw: migrating ==================
-- rename_column(:scaptimony_arf_report_raws, :raw, :bzip_data)rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"ok", :duration_ms=>"106"}, :pulp_auth=>{:status=>"ok", :duration_ms=>"111"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/6.0.24</h3></body></html> (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'

Comment 2 Lukas Pramuk 2016-05-23 11:10:21 UTC
/var/log/foreman-installer/satellite.log


[ INFO 2016-05-19 18:57:10 main] Upgrade Step: db_seed...
[ERROR 2016-05-19 18:57:24 main] Run `rake db:migrate` to update your database then try again.
You have 10 pending migrations:
  20151009084350 DropPtables
  20151019174035 RenameDomainHostCount
  20151025120534 AddHiddenValueToLookupKey
  20151104100257 AddHostsCountToHostgroup
  20151109152507 AddHostStatusHostIdIndex
  20151120153254 DeleteBootableInterface
  20151220093801 RemoveSpacesFromSmartVariableKey
  20160201131211 AddExpiredLogsToSmartProxy
  20160215143900 AddSubnetDomainRelationConstraints
  20160404074723 DowncaseDisplayTypes

[ERROR 2016-05-19 18:57:24 main] Upgrade step db_seed failed. Check logs for more information.
[DEBUG 2016-05-19 18:57:24 main] Exit with status code: 1 (signal was 1)

Comment 4 Lukas Pramuk 2016-05-23 12:41:30 UTC
Some backend services are not running:
 - caused by passenger versions inconsistency https://bugzilla.redhat.com/show_bug.cgi?id=1336831

So we can expect that fix for #1336831 will fix db_migrate step too.

Comment 6 Brad Buckingham 2016-05-23 16:43:21 UTC
The 'db_migrate' error appears to be a duplicate of bug 1336941.  I am going to run a test with the fix that is currently under review upstream and will close this BZ as a duplicate if it solves the issue on RHEL6.

Comment 7 Brad Buckingham 2016-05-24 13:12:01 UTC
With the fix for bug 1336941, I did not see the 'db_migrate' error on RHEL6 upgrade; therefore, I am going to close this one as a duplicate.  Please re-open if the issue persists after the next snap available later this week.

*** This bug has been marked as a duplicate of bug 1336941 ***


Note You need to log in before you can comment on or make changes to this bug.