Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1338737

Summary: Upgrade step db_migrate failed on RHEL6
Product: Red Hat Satellite Reporter: Lukas Pramuk <lpramuk>
Component: UpgradesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Lukas Pramuk <lpramuk>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-24 13:12:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335807    

Description Lukas Pramuk 2016-05-23 11:07:07 UTC
Description of problem:
Upgrade step db_migrate failed on RHEL6


Version-Release number of selected component (if applicable):
Sat6.2.0-Snap12

How reproducible:
always

Steps to Reproduce:
# satellite-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Could not evaluate: Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Failed to call refresh: Proxy <SAT_FQDN> cannot be registered (Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A): N/A
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<SAT_FQDN>]: Proxy <SAT_FQDN> cannot be registered (Proxy <SAT_FQDN> cannot be registered (500 Internal Server Error): N/A): N/A
Installing             Done                                               [100%] [....................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
/usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `error': wrong number of arguments (49 for 1) (ArgumentError)
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `send'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:108:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:107:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:98:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/logger.rb:103:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:28:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:58:in `fail_and_exit'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:52:in `upgrade_step'
	from /usr/share/katello-installer-base/hooks/post/30-upgrade.rb:65:in `load'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:51:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/hooking.rb:49:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:412:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `call'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/exit_handler.rb:26:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:162:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:411:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:149:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.7.3/lib/kafo/kafo_configure.rb:156:in `run'
	from /usr/sbin/foreman-installer:12

Actual results:
db_migrate failed due to not running services

Expected results:
db_migrate succeeds

Additional info:
After the db_seed failed due to failed db_migrate, I was able to recover by just reruning db_migrate (services were started before db_seed)

Comment 1 Lukas Pramuk 2016-05-23 11:09:04 UTC
/var/log/foreman/db_migrate.log


== 20150817102538 AddDelayAttributes: migrating ===============================
-- add_column(:foreman_tasks_tasks, :start_at, :datetime, {:index=>true, :default=>nil, :null=>true})
   -> 0.0009s
-- add_column(:foreman_tasks_tasks, :start_before, :datetime, {:index=>true, :default=>nil, :null=>true})rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"400 Bad Request"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 400 Bad Request  (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'


== 20150930183738 MigrateContentHosts: migrating ==============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/6.0.24</h3></body></html> (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"4"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'


== 20150114210634 RenameScaptimonyArfReportRawRaw: migrating ==================
-- rename_column(:scaptimony_arf_report_raws, :raw, :bzip_data)rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"ok", :duration_ms=>"106"}, :pulp_auth=>{:status=>"ok", :duration_ms=>"111"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found <html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/6.0.24</h3></body></html> (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'

Comment 2 Lukas Pramuk 2016-05-23 11:10:21 UTC
/var/log/foreman-installer/satellite.log


[ INFO 2016-05-19 18:57:10 main] Upgrade Step: db_seed...
[ERROR 2016-05-19 18:57:24 main] Run `rake db:migrate` to update your database then try again.
You have 10 pending migrations:
  20151009084350 DropPtables
  20151019174035 RenameDomainHostCount
  20151025120534 AddHiddenValueToLookupKey
  20151104100257 AddHostsCountToHostgroup
  20151109152507 AddHostStatusHostIdIndex
  20151120153254 DeleteBootableInterface
  20151220093801 RemoveSpacesFromSmartVariableKey
  20160201131211 AddExpiredLogsToSmartProxy
  20160215143900 AddSubnetDomainRelationConstraints
  20160404074723 DowncaseDisplayTypes

[ERROR 2016-05-19 18:57:24 main] Upgrade step db_seed failed. Check logs for more information.
[DEBUG 2016-05-19 18:57:24 main] Exit with status code: 1 (signal was 1)

Comment 4 Lukas Pramuk 2016-05-23 12:41:30 UTC
Some backend services are not running:
 - caused by passenger versions inconsistency https://bugzilla.redhat.com/show_bug.cgi?id=1336831

So we can expect that fix for #1336831 will fix db_migrate step too.

Comment 6 Brad Buckingham 2016-05-23 16:43:21 UTC
The 'db_migrate' error appears to be a duplicate of bug 1336941.  I am going to run a test with the fix that is currently under review upstream and will close this BZ as a duplicate if it solves the issue on RHEL6.

Comment 7 Brad Buckingham 2016-05-24 13:12:01 UTC
With the fix for bug 1336941, I did not see the 'db_migrate' error on RHEL6 upgrade; therefore, I am going to close this one as a duplicate.  Please re-open if the issue persists after the next snap available later this week.

*** This bug has been marked as a duplicate of bug 1336941 ***