Description of problem: Upgrade step db_migrate failed on RHEL6 Version-Release number of selected component (if applicable): Sat6.2.0-Snap12 How reproducible: always Steps to Reproduce: # satellite-installer --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: start_httpd... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: migrate_foreman... Upgrade Step: migrate_gutterball... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Could not evaluate: Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Failed to call refresh: Proxy <SAT_FQDN> cannot be registered (Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A): N/A /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Proxy <SAT_FQDN> cannot be registered (Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A): N/A Installing Done [100%] [....................................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information. /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `error': wrong number of arguments (49 for 1) (ArgumentError) from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `send' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `each' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `each' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `dump_buffer' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:98:in `dump_errors' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:103:in `dump_errors' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:28:in `exit' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit' from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:58:in `fail_and_exit' from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:52:in `upgrade_step' from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:65:in `load' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `instance_exec' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `execute' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:51:in `execute' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `each' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `execute' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:412:in `run_installation' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `call' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `exit' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:411:in `run_installation' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:149:in `execute' from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run' from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run' from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:156:in `run' from /usr/sbin/foreman-installer:12 Actual results: db_migrate failed due to not running services Expected results: db_migrate succeeds Additional info: After the db_seed failed due to failed db_migrate, I was able to recover by just reruning db_migrate (services were started before db_seed)
/var/log/foreman/db_migrate.log == 20150817102538 AddDelayAttributes: migrating =============================== -- add_column(:foreman_tasks_tasks, :start_at, :datetime, {:index=>true, :default=>nil, :null=>true}) -> 0.0009s -- add_column(:foreman_tasks_tasks, :start_before, :datetime, {:index=>true, :default=>nil, :null=>true})rake aborted! StandardError: An error has occurred, this and all later migrations canceled: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"400 Bad Request"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 400 Bad Request (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate' == 20150930183738 MigrateContentHosts: migrating ============================== rake aborted! StandardError: An error has occurred, this and all later migrations canceled: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/6.0.24</h3></body></html> (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"4"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection' == 20150114210634 RenameScaptimonyArfReportRawRaw: migrating ================== -- rename_column(:scaptimony_arf_report_raws, :raw, :bzip_data)rake aborted! StandardError: An error has occurred, this and all later migrations canceled: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"ok", :duration_ms=>"106"}, :pulp_auth=>{:status=>"ok", :duration_ms=>"111"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/6.0.24</h3></body></html> (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'
/var/log/foreman-installer/satellite.log [ INFO 2016-05-19 18:57:10 main] Upgrade Step: db_seed... [ERROR 2016-05-19 18:57:24 main] Run `rake db:migrate` to update your database then try again. You have 10 pending migrations: 20151009084350 DropPtables 20151019174035 RenameDomainHostCount 20151025120534 AddHiddenValueToLookupKey 20151104100257 AddHostsCountToHostgroup 20151109152507 AddHostStatusHostIdIndex 20151120153254 DeleteBootableInterface 20151220093801 RemoveSpacesFromSmartVariableKey 20160201131211 AddExpiredLogsToSmartProxy 20160215143900 AddSubnetDomainRelationConstraints 20160404074723 DowncaseDisplayTypes [ERROR 2016-05-19 18:57:24 main] Upgrade step db_seed failed. Check logs for more information. [DEBUG 2016-05-19 18:57:24 main] Exit with status code: 1 (signal was 1)
Some backend services are not running: - caused by passenger versions inconsistency https://bugzilla.redhat.com/show_bug.cgi?id=1336831 So we can expect that fix for #1336831 will fix db_migrate step too.
The 'db_migrate' error appears to be a duplicate of bug 1336941. I am going to run a test with the fix that is currently under review upstream and will close this BZ as a duplicate if it solves the issue on RHEL6.
With the fix for bug 1336941, I did not see the 'db_migrate' error on RHEL6 upgrade; therefore, I am going to close this one as a duplicate. Please re-open if the issue persists after the next snap available later this week. *** This bug has been marked as a duplicate of bug 1336941 ***