Hide Forgot
Description of problem: I was trying to perform upgrade from [sat6.1.9 -> Sat6.2 snap12] and it failed at step: ~]# satellite-installer --scenario satellite --upgrade --force Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: start_httpd... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: migrate_foreman... Upgrade Step: migrate_gutterball... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Could not evaluate: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Failed to call refresh: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A Installing Done [100%] [............................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information. I think upgrade failed at db:migrate while running installer and later it failed at db_seed Version-Release number of selected component (if applicable): Sat[6.1.9 ->6.2 snap12] How reproducible: Steps to Reproduce: 1. install Satellite 6.1 2. update packages to Satellite 6.2 3. satellite-installer --scenario satellite --upgrade --force Actual results: I think upgrade failed at db:migrate while running installer and later it failed at db_seed Expected results: upgrade should succeed without any error on stdout or in logs. Additional info:
Please note that as per staellite.log, upgrade failed first at db:migrate. [ERROR 2016-05-19 05:47:11 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] [ERROR 2016-05-19 05:47:11 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] [ERROR 2016-05-19 05:47:57 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ERROR 2016-05-19 05:47:57 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] And I see another issue with proxy as per below error: [ERROR 2016-05-19 06:01:31 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Could not evaluate: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A [ERROR 2016-05-19 06:01:32 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Failed to call refresh: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A [ERROR 2016-05-19 06:01:32 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A
Created attachment 1159352 [details] foreman-debug logs
foreman-rake db:migrate failed with below error. ~]# foreman-rake db:migrate --trace ** Invoke db:migrate (first_time) ** Invoke environment (first_time) ** Execute environment ** Invoke db:load_config (first_time) ** Execute db:load_config ** Execute db:migrate == 20150612135546 CreateHostStatus: migrating ================================= -- create_table(:host_status) -> 0.0091s -- add_index(:host_status, :host_id) -> 0.0043s -- add_foreign_key("host_status", "hosts", {:name=>"host_status_hosts_host_id_fk", :column=>"host_id"}) -> 0.0034s -- add_column(:hosts, :global_status, :integer, {:default=>0, :null=>false}) -> 0.0396s rake aborted! StandardError: An error has occurred, this and all later migrations canceled: PG::Error: ERROR: relation "foreman_openscap_assets" does not exist LINE 5: WHERE a.attrelid = '"foreman_openscap_assets"... ^ : SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = '"foreman_openscap_assets"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum
Hi Sachin, I think this bug may be raising multiple issues. The first is the failure on db:migrate/db:seed. In our testing, I have seen a failure during db:migrate; however, it was related to bug 1336941. I'd like to get Shlomi to take a peak at the error you see as it may be new to the snap. The second is the failure on registering with the proxy. For that one, I believe the issue is related to bug 1336831. One way to confirm if this is the case is to perform the workaround mentioned in bug 1336831#1.
Hi Shlomi, can you take a look at the upgrade error raised in comment #3? Could this be related to a recent change in snap 12?
Thanks Brad. Yeah.. agreed the db:migrate issue is similar to what you'r getting with bug 1336941. thanks And, I tried the workaround given in https://bugzilla.redhat.com/show_bug.cgi?id=1336831#c1 with snap12 and this addresses the issue related to failure on registering with the proxy. Now installer failed with db:seed only: ~]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: start_httpd... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: migrate_foreman... Upgrade Step: migrate_gutterball... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] Installing Done [100%] [............................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information.
And please note that this issue is reproducible if we performed upgrade with following steps: 1. Disable sat6.1 repo 2. Set/enable sat6.2 repo 3. yum install satellite -y 4. yum update -y 5. satellite-installer --scenario satellite --upgrade
Reproducible with upgrade from sat6.1.9 -> Sat6.2 snap13 [root@qe-sat6-upgrade-rhel7 ~]# foreman-rake db:migrate --trace ** Invoke db:migrate (first_time) ** Invoke environment (first_time) ** Execute environment ** Invoke db:load_config (first_time) ** Execute db:load_config ** Execute db:migrate == 20150612135546 CreateHostStatus: migrating ================================= -- create_table(:host_status) -> 0.0090s -- add_index(:host_status, :host_id) -> 0.0043s -- add_foreign_key("host_status", "hosts", {:name=>"host_status_hosts_host_id_fk", :column=>"host_id"}) -> 0.0035s -- add_column(:hosts, :global_status, :integer, {:default=>0, :null=>false}) -> 0.0744s rake aborted! StandardError: An error has occurred, this and all later migrations canceled: PG::Error: ERROR: relation "foreman_openscap_assets" does not exist LINE 5: WHERE a.attrelid = '"foreman_openscap_assets"... ^ : SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = '"foreman_openscap_assets"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `block in exec_no_cache'
console logs while upgrading sat6.1.9 -> Sat6.2 snap13 on rhel7 [root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: start_httpd... Upgrade Step: start_qpidd... Upgrade Step: start_pulp... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... Upgrade Step: migrate_gutterball... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca Installing Done [100%] [..................................................................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information. [r
[ERROR 2016-05-26 01:35:38 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ERROR 2016-05-26 01:35:38 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in `process_callback' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in `block in process_events' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in `block in queued_events' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `each' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `queued_events' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in `process_events' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/benchmark.rb:296:in `realtime' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse' [ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate' [ INFO 2016-05-26 01:35:38 main] /usr/share/gems/gems/kafo-0.7.3/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:25:in `evaluate_with_trigger' [
This looks like a duplicate of 1337462 ?
It's not entirely a duplicate but the fix for 1337462 should fix this as well.
Performed upgrade from sat6.1.9 -> sat6.2 snap14 on rhel7: Installer still failed at "Upgrade step db_seed failed:" Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: start_httpd... Upgrade Step: start_qpidd... Upgrade Step: start_pulp... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] Installing Done [100%] [..................................................................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information. I ran db:migrate manually and got this error: undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing' will file a separate issue to track db:migrate issue.
Filed another bz 1341974 for undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>
This is tracked by bz 1341860. I'm keeping this open to verify openscap update once it's ready.
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7. Upgrade was successfully completed and I did not get issue related to db:migrate and db:seed. Also all the reported issues in this bz are fixed. Please see the screenshot of first successful upgrade. thank you for fixing all related issues.
Created attachment 1164365 [details] upgrade completed
Just to correct: Verification is done with upgrade from sat6.1.9 -> sat6.2 GA snap14.1 on rhel7. Rpm version is 14.2 satellite-6.2.0-14.2.el7sat.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501