Created attachment 1158533 [details] Satellite upgrade installer log Description of problem: Attempting to upgrade to an early version of Satellite 6.2 Beta (internal build) from 6.1 is resulting in an error during foreman_migrate. Version-Release number of selected component (if applicable): Satellite 6.2 Snap 11 How reproducible: Always Steps to Reproduce: 1. install Satellite 6.1 2. update packages to Satellite 6.2 3. patch system to include fixes for bug 1334961, bug 1335675, bug 1335699, bug 1335700, bug 1335967, bug 1336831 4. satellite-installer --scenario satellite --upgrade --force Actual results: # satellite-installer --scenario satellite --upgrade --force Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: start_httpd... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: migrate_foreman... Upgrade Step: migrate_gutterball... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] Installing Done [100%] [........................................................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information. Expected results: No error during the migrate_foreman step. Additional info: Note: see attached log for complete installer run. In the /var/log/foreman-installer/satellite.log, the following is observed: ...[snip]... Some backend services are not running: {:status=>"FAIL", :services=>{:pulp=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :pulp_auth=>{:status=>"FAIL", :message=>"Skipped pulp_auth check after failed pulp check"}, :candlepin=>{:status=>"FAIL", :message=>"404 Resource Not Found"}, :candlepin_auth=>{:status=>"FAIL", :message=>"Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found (GET /candlepin/status)"}, :foreman_tasks=>{:status=>"ok", :duration_ms=>"5"}}}/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.28/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up' ...[snip]... /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.28/db/migrate/20150930183738_migrate_content_hosts.rb:297:in `up' ...[snip]...
To do some investigation, I updated /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb to perform the following: def migrate_foreman Kafo::Helpers.execute('foreman-rake -- config -k use_pulp_oauth -v true') exit 1 Kafo::Helpers.execute('foreman-rake db:migrate') ... ... This caused the upgrade to abort and I was able to confirm the 404 behavior using: # curl -s -k -u admin:EoxjDuoYdKmMZyPiLhtB94Kz3o82nivu https://localhost/pulp/api/v2//status/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /pulp/api/v2//status/ was not found on this server.</p> </body></html> It may be unrelated; however, I also looked over the /etc/httpd/conf.d/* to compare the upgraded system against a 'satellite 6.2' install. I do see a lot of differences in the *.conf files (e.g. pulp.conf). Those differences may be part of the reason for the 404.
*** Bug 1338737 has been marked as a duplicate of this bug. ***
Console logs while upgrading sat6.1.9 -> Sat6.2 snap13 on rhel7 still shows similar error as reported in bz description. I'll move this verified only with successful installer. [root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: start_httpd... Upgrade Step: start_qpidd... Upgrade Step: start_pulp... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... Upgrade Step: migrate_gutterball... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca Installing Done [100%] [..................................................................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information.
The error in comment 6 looks very similar to an error that I encountered yesterday. It was raised with Eric and should have been resolved. Eric, can you take a look? This is regarding the following error during db:seed: Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
Sachin, the error reported in comment 6 is different than the original failure raised by this bug. That error is covered by bug 1339901.
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7. Upgrade was successfully completed. I don't see any issue with migrate_foreman step. SATELLITE UPGRADE started at: Fri Jun 3 02:18:11 2016 [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] run: satellite-installer --scenario satellite --upgrade [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrading... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: stop_services... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_databases... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: update_http_conf... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_pulp... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_pulp_httpd_conf... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_httpd... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_qpidd... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_pulp... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_candlepin... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_tomcat... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_katello_settings_file... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_foreman... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_nodes_distributors... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: Running installer... Done [100%] []Installing Done [100%] [] [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: The full log is at /var/log/foreman-installer/satellite.log [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: restart_services... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: db_seed... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_package_groups (this may take a while) ... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_rpms (this may take a while) ... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_distributions (this may take a while) ... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_puppet_modules (this may take a while) ... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_subscriptions (this may take a while) ... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: elasticsearch_message... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Elasticsearch has been removed as a dependency. The database files can be removed manually with #rm -rf /var/lib/elasticsearch. Some packages are no longer needed and can be removed: #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4 [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_docker_v1_content (this may take a while) ... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_gutterball... [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Katello upgrade completed! [qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: SATELLITE UPGRADE finished at: Fri Jun 3 03:07:19 2016
Just to correct: Verification is done with upgrade from sat6.1.9 -> sat6.2 GA snap14.1 on rhel7. Rpm version is 14.2 satellite-6.2.0-14.2.el7sat.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501