Please note that as per staellite.log, upgrade failed first at db:migrate.


[ERROR 2016-05-19 05:47:11 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[ERROR 2016-05-19 05:47:11 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[ERROR 2016-05-19 05:47:57 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-05-19 05:47:57 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]


And I see another issue with proxy as per below error:

[ERROR 2016-05-19 06:01:31 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Could not evaluate: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A
[ERROR 2016-05-19 06:01:32 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Failed to call refresh: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A
[ERROR 2016-05-19 06:01:32 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A

Created attachment 1159352 [details]
foreman-debug logs

foreman-rake db:migrate failed with below error.

~]# foreman-rake db:migrate --trace 
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
== 20150612135546 CreateHostStatus: migrating =================================
-- create_table(:host_status)
   -> 0.0091s
-- add_index(:host_status, :host_id)
   -> 0.0043s
-- add_foreign_key("host_status", "hosts", {:name=>"host_status_hosts_host_id_fk", :column=>"host_id"})
   -> 0.0034s
-- add_column(:hosts, :global_status, :integer, {:default=>0, :null=>false})
   -> 0.0396s
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "foreman_openscap_assets" does not exist
LINE 5:                WHERE a.attrelid = '"foreman_openscap_assets"...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"foreman_openscap_assets"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

Hi Sachin,

I think this bug may be raising multiple issues.

The first is the failure on db:migrate/db:seed.  In our testing, I have seen a failure during db:migrate; however, it was related to bug 1336941.  I'd like to get Shlomi to take a peak at the error you see as it may be new to the snap.

The second is the failure on registering with the proxy.  For that one, I believe the issue is related to bug 1336831.  One way to confirm if this is the case is to perform the workaround mentioned in bug 1336831#1.

Hi Shlomi,  can you take a look at the upgrade error raised in comment #3?  Could this be related to a recent change in snap 12?

Thanks Brad. Yeah.. agreed the db:migrate issue is similar to what you'r getting with bug 1336941. thanks

And, I tried the workaround given in https://bugzilla.redhat.com/show_bug.cgi?id=1336831#c1 with snap12 and this addresses the issue related to failure on registering with the proxy. Now installer failed with db:seed only:

~]# satellite-installer --scenario satellite --upgrade 
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [............................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.

And please note that this issue is reproducible if we performed upgrade with following steps:

1. Disable sat6.1 repo 
2. Set/enable sat6.2 repo
3. yum install satellite -y
4. yum update -y
5. satellite-installer --scenario satellite --upgrade

Reproducible with upgrade from sat6.1.9 -> Sat6.2 snap13

[root@qe-sat6-upgrade-rhel7 ~]# foreman-rake db:migrate --trace 
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
== 20150612135546 CreateHostStatus: migrating =================================
-- create_table(:host_status)
   -> 0.0090s
-- add_index(:host_status, :host_id)
   -> 0.0043s
-- add_foreign_key("host_status", "hosts", {:name=>"host_status_hosts_host_id_fk", :column=>"host_id"})
   -> 0.0035s
-- add_column(:hosts, :global_status, :integer, {:default=>0, :null=>false})
   -> 0.0744s
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "foreman_openscap_assets" does not exist
LINE 5:                WHERE a.attrelid = '"foreman_openscap_assets"...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"foreman_openscap_assets"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `block in exec_no_cache'

console logs while upgrading sat6.1.9 -> Sat6.2 snap13 on rhel7

[root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
Installing             Done                                               [100%] [..................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
[r

[ERROR 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in `process_callback'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in `block in process_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in `block in queued_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `each'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `queued_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in `process_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/benchmark.rb:296:in `realtime'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
[ INFO 2016-05-26 01:35:38 main] /usr/share/gems/gems/kafo-0.7.3/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:25:in `evaluate_with_trigger'
[

This looks like a duplicate of 1337462 ?

It's not entirely a duplicate but the fix for 1337462 should fix this as well.

Performed upgrade from sat6.1.9 -> sat6.2 snap14 on rhel7:

Installer still failed at "Upgrade step db_seed failed:"

Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...

Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [..................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.


I ran db:migrate manually and got this error:

undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing'


will file a separate issue to track db:migrate issue.

Filed another bz 1341974 for undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>

This is tracked by bz 1341860. I'm keeping this open to verify openscap update once it's ready.

Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7.

Upgrade was successfully completed and I did not get issue related to db:migrate and db:seed. Also all the reported issues in this bz are fixed.

Please see the screenshot of first successful upgrade. thank you for fixing all related issues.

Created attachment 1164365 [details]
upgrade completed

Just to correct: Verification is done with upgrade from sat6.1.9 -> sat6.2 GA snap14.1 on rhel7. Rpm version is 14.2

satellite-6.2.0-14.2.el7sat.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501