Description of problem: upgrade from sat6.1.11 -> sat6.2.7 snap7 on rhel7 failed at migrate_foreman step: [Sat_SERVER] out: rake aborted! [Sat_SERVER] out: StandardError: An error has occurred, this and all later migrations canceled: [Sat_SERVER] out: [Sat_SERVER] out: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"45"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"84"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"9"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in `block in execute_migration_in_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `block in ddl_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `ddl_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in `execute_migration_in_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in `block in migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `each' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in `up' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in `block (2 levels) in <top (required)>' [Sat_SERVER] out: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"45"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"84"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"9"}}} [Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in `block in execute_migration_in_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `block in ddl_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `ddl_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in `execute_migration_in_transaction' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in `block in migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `each' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in `up' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in `migrate' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in `block (2 levels) in <top (required)>' [Sat_SERVER] out: Tasks: TOP => db:migrate [Sat_SERVER] out: (See full trace by running task with --trace) [Sat_SERVER] out: == 20150930183738 MigrateContentHosts: migrating Version-Release number of selected component (if applicable): upgrade sat6.1.11 -> sat6.2.7 snap2 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: [Sat_SERVER] out: rake aborted! [Sat_SERVER] out: StandardError: An error has occurred, this and all later migrations canceled: [Sat_SERVER] out: [Sat_SERVER] out: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"45"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"84"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"9"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up' [Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration' Expected results: upgrade should succeed. Additional info:
Created attachment 1242474 [details] installer logs from console
Created attachment 1242476 [details] foreman-debug
The underlying cause is Satellite uses /etc/httpd/conf.d/pulp.conf, whereas capsules do not. On a capsule, the RPM package contains a pulp.conf that has conflicting values so we need to delete it (that was BZ1368085), but only on capsules.
FailedQA. @Sat6.2.7 Snap2 It is still failing Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... true rake aborted! StandardError: An error has occurred, this and all later migrations canceled: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"93"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"101"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"7"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up' # rpm -q katello-installer-base katello-installer-base-3.0.0.70-1.el7sat.noarch (!=Fixed In Version) >>> I don't see that the fix landed downstream
Verified with sat6.2.7 snap3 I successfully upgraded sat6.1.11 -> sat6.2.7 snap3. Please see the attached logs for more info.
Created attachment 1243870 [details] sat_installer.log on successful upgrade
This was delivered in satellite 6.2.7 (https://access.redhat.com/errata/RHBA-2017:0197)