Bug 1414780 - Upgrade from sat6.1 -> sat6.2.7 snap2 failed at migrate_foreman step
Summary: Upgrade from sat6.1 -> sat6.2.7 snap2 failed at migrate_foreman step
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.2.7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Sachin Ghai
URL:
Whiteboard:
Depends On:
Blocks: Sat6_Upgrades
TreeView+ depends on / blocked
 
Reported: 2017-01-19 12:29 UTC by Sachin Ghai
Modified: 2019-09-26 17:30 UTC (History)
7 users (show)

Fixed In Version: katello-installer-base-3.0.0.71-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-26 15:55:55 UTC
Target Upstream Version:


Attachments (Terms of Use)
installer logs from console (70.45 KB, text/plain)
2017-01-19 12:37 UTC, Sachin Ghai
no flags Details
foreman-debug (1.95 MB, application/x-xz)
2017-01-19 12:42 UTC, Sachin Ghai
no flags Details
sat_installer.log on successful upgrade (83.69 KB, text/plain)
2017-01-24 10:14 UTC, Sachin Ghai
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 18132 0 None None None 2017-01-19 15:13:06 UTC

Description Sachin Ghai 2017-01-19 12:29:21 UTC
Description of problem:
upgrade from sat6.1.11 -> sat6.2.7 snap7 on rhel7 failed at migrate_foreman step:


[Sat_SERVER] out: rake aborted!
[Sat_SERVER] out: StandardError: An error has occurred, this and all later migrations canceled:
[Sat_SERVER] out: 
[Sat_SERVER] out: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"45"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"84"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"9"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in `block in execute_migration_in_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `block in ddl_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `ddl_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in `execute_migration_in_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in `block in migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `each'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in `up'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in `block (2 levels) in <top (required)>'
[Sat_SERVER] out: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"45"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"84"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"9"}}}
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:579:in `block (2 levels) in migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:578:in `block in migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in `with_connection'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:577:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:752:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:992:in `block in execute_migration_in_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `block in ddl_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:1038:in `ddl_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:991:in `execute_migration_in_transaction'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:953:in `block in migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `each'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:949:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:807:in `up'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:785:in `migrate'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/databases.rake:34:in `block (2 levels) in <top (required)>'
[Sat_SERVER] out: Tasks: TOP => db:migrate
[Sat_SERVER] out: (See full trace by running task with --trace)
[Sat_SERVER] out: == 20150930183738 MigrateContentHosts: migrating 
Version-Release number of selected component (if applicable):
upgrade sat6.1.11 -> sat6.2.7 snap2

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
[Sat_SERVER] out: rake aborted!
[Sat_SERVER] out: StandardError: An error has occurred, this and all later migrations canceled:
[Sat_SERVER] out: 
[Sat_SERVER] out: Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"45"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"84"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"9"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up'
[Sat_SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/migration.rb:598:in `exec_migration'

Expected results:
upgrade should succeed.

Additional info:

Comment 3 Sachin Ghai 2017-01-19 12:37:31 UTC
Created attachment 1242474 [details]
installer logs from console

Comment 4 Sachin Ghai 2017-01-19 12:42:37 UTC
Created attachment 1242476 [details]
foreman-debug

Comment 6 Stephen Benjamin 2017-01-19 15:20:53 UTC
The underlying cause is Satellite uses /etc/httpd/conf.d/pulp.conf, whereas capsules do not.  

On a capsule, the RPM package contains a pulp.conf that has conflicting values so we need to delete it (that was BZ1368085), but only on capsules.

Comment 8 Lukas Pramuk 2017-01-23 10:23:18 UTC
FailedQA.

@Sat6.2.7 Snap2

It is still failing

Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
true

rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"ok", :duration_ms=>"93"}, :candlepin_auth=>{:status=>"ok", :duration_ms=>"101"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"7"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.93/db/migrate/20150930183738_migrate_content_hosts.rb:343:in `up'

# rpm -q  katello-installer-base
katello-installer-base-3.0.0.70-1.el7sat.noarch (!=Fixed In Version)

>>> I don't see that the fix landed downstream

Comment 10 Sachin Ghai 2017-01-24 10:10:57 UTC
Verified with sat6.2.7 snap3

I successfully upgraded sat6.1.11 -> sat6.2.7 snap3. Please see the attached logs for more info.

Comment 11 Sachin Ghai 2017-01-24 10:14:22 UTC
Created attachment 1243870 [details]
sat_installer.log on successful upgrade

Comment 12 Bryan Kearney 2017-01-26 15:55:55 UTC
This was delivered in satellite 6.2.7 (https://access.redhat.com/errata/RHBA-2017:0197)


Note You need to log in before you can comment on or make changes to this bug.