Bug 1337493 - upgrade[6.1.9 ->6.2 snap12] at step: db_seed failed
Summary: upgrade[6.1.9 ->6.2 snap12] at step: db_seed failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Shlomi Zadok
QA Contact: Sachin Ghai
URL:
Whiteboard:
Depends On: 1337462 1341860
Blocks: 1335807
TreeView+ depends on / blocked
 
Reported: 2016-05-19 10:17 UTC by Sachin Ghai
Modified: 2019-09-26 17:33 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:24:36 UTC
Target Upstream Version:


Attachments (Terms of Use)
foreman-debug logs (1.03 MB, application/x-xz)
2016-05-19 10:24 UTC, Sachin Ghai
no flags Details
upgrade completed (114.81 KB, image/png)
2016-06-03 07:50 UTC, Sachin Ghai
no flags Details

Description Sachin Ghai 2016-05-19 10:17:50 UTC
Description of problem:
I was trying to perform upgrade from [sat6.1.9 -> Sat6.2 snap12] and it failed at step:

~]# satellite-installer --scenario satellite --upgrade --force
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Could not evaluate: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Failed to call refresh: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A
Installing             Done                                               [100%] [............................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.


I think upgrade failed at db:migrate while running installer and later it failed at db_seed


Version-Release number of selected component (if applicable):

Sat[6.1.9 ->6.2 snap12] 

How reproducible:


Steps to Reproduce:
1. install Satellite 6.1
2. update packages to Satellite 6.2
3. satellite-installer --scenario satellite --upgrade --force

Actual results:
I think upgrade failed at db:migrate while running installer and later it failed at db_seed

Expected results:
upgrade should succeed without any error on stdout or in logs.

Additional info:

Comment 1 Sachin Ghai 2016-05-19 10:20:21 UTC
Please note that as per staellite.log, upgrade failed first at db:migrate.


[ERROR 2016-05-19 05:47:11 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[ERROR 2016-05-19 05:47:11 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[ERROR 2016-05-19 05:47:57 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-05-19 05:47:57 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]


And I see another issue with proxy as per below error:

[ERROR 2016-05-19 06:01:31 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Could not evaluate: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A
[ERROR 2016-05-19 06:01:32 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Failed to call refresh: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A
[ERROR 2016-05-19 06:01:32 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com]: Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (Proxy qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com cannot be registered (500 Internal Server Error): N/A): N/A

Comment 2 Sachin Ghai 2016-05-19 10:24:28 UTC
Created attachment 1159352 [details]
foreman-debug logs

Comment 3 Sachin Ghai 2016-05-19 10:39:43 UTC
foreman-rake db:migrate failed with below error.

~]# foreman-rake db:migrate --trace 
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
== 20150612135546 CreateHostStatus: migrating =================================
-- create_table(:host_status)
   -> 0.0091s
-- add_index(:host_status, :host_id)
   -> 0.0043s
-- add_foreign_key("host_status", "hosts", {:name=>"host_status_hosts_host_id_fk", :column=>"host_id"})
   -> 0.0034s
-- add_column(:hosts, :global_status, :integer, {:default=>0, :null=>false})
   -> 0.0396s
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "foreman_openscap_assets" does not exist
LINE 5:                WHERE a.attrelid = '"foreman_openscap_assets"...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"foreman_openscap_assets"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

Comment 5 Brad Buckingham 2016-05-20 02:12:46 UTC
Hi Sachin,

I think this bug may be raising multiple issues.

The first is the failure on db:migrate/db:seed.  In our testing, I have seen a failure during db:migrate; however, it was related to bug 1336941.  I'd like to get Shlomi to take a peak at the error you see as it may be new to the snap.

The second is the failure on registering with the proxy.  For that one, I believe the issue is related to bug 1336831.  One way to confirm if this is the case is to perform the workaround mentioned in bug 1336831#1.

Comment 6 Brad Buckingham 2016-05-20 02:16:19 UTC
Hi Shlomi,  can you take a look at the upgrade error raised in comment #3?  Could this be related to a recent change in snap 12?

Comment 7 Sachin Ghai 2016-05-20 09:24:25 UTC
Thanks Brad. Yeah.. agreed the db:migrate issue is similar to what you'r getting with bug 1336941. thanks

And, I tried the workaround given in https://bugzilla.redhat.com/show_bug.cgi?id=1336831#c1 with snap12 and this addresses the issue related to failure on registering with the proxy. Now installer failed with db:seed only:

~]# satellite-installer --scenario satellite --upgrade 
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [............................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.

Comment 9 Sachin Ghai 2016-05-20 09:26:19 UTC
And please note that this issue is reproducible if we performed upgrade with following steps:

1. Disable sat6.1 repo 
2. Set/enable sat6.2 repo
3. yum install satellite -y
4. yum update -y
5. satellite-installer --scenario satellite --upgrade

Comment 12 Sachin Ghai 2016-05-26 05:52:56 UTC
Reproducible with upgrade from sat6.1.9 -> Sat6.2 snap13

[root@qe-sat6-upgrade-rhel7 ~]# foreman-rake db:migrate --trace 
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
== 20150612135546 CreateHostStatus: migrating =================================
-- create_table(:host_status)
   -> 0.0090s
-- add_index(:host_status, :host_id)
   -> 0.0043s
-- add_foreign_key("host_status", "hosts", {:name=>"host_status_hosts_host_id_fk", :column=>"host_id"})
   -> 0.0035s
-- add_column(:hosts, :global_status, :integer, {:default=>0, :null=>false})
   -> 0.0744s
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "foreman_openscap_assets" does not exist
LINE 5:                WHERE a.attrelid = '"foreman_openscap_assets"...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"foreman_openscap_assets"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `block in exec_no_cache'

Comment 13 Sachin Ghai 2016-05-26 05:59:23 UTC
console logs while upgrading sat6.1.9 -> Sat6.2 snap13 on rhel7

[root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
Installing             Done                                               [100%] [..................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
[r

Comment 14 Sachin Ghai 2016-05-26 07:40:00 UTC
[ERROR 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in `process_callback'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in `block in process_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in `block in queued_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `each'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `queued_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in `process_events'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/benchmark.rb:296:in `realtime'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
[ INFO 2016-05-26 01:35:38 main] /usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
[ INFO 2016-05-26 01:35:38 main] /usr/share/gems/gems/kafo-0.7.3/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:25:in `evaluate_with_trigger'
[

Comment 17 Ohad Levy 2016-05-30 10:19:26 UTC
This looks like a duplicate of 1337462 ?

Comment 18 Marek Hulan 2016-05-30 11:13:59 UTC
It's not entirely a duplicate but the fix for 1337462 should fix this as well.

Comment 19 Sachin Ghai 2016-06-02 07:44:19 UTC
Performed upgrade from sat6.1.9 -> sat6.2 snap14 on rhel7:

Installer still failed at "Upgrade step db_seed failed:"

Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...

Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [..................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.


I ran db:migrate manually and got this error:

undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing'


will file a separate issue to track db:migrate issue.

Comment 20 Sachin Ghai 2016-06-02 07:54:57 UTC
Filed another bz 1341974 for undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>

Comment 21 Marek Hulan 2016-06-02 09:45:33 UTC
This is tracked by bz 1341860. I'm keeping this open to verify openscap update once it's ready.

Comment 23 Sachin Ghai 2016-06-03 07:49:38 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7.

Upgrade was successfully completed and I did not get issue related to db:migrate and db:seed. Also all the reported issues in this bz are fixed.

Please see the screenshot of first successful upgrade. thank you for fixing all related issues.

Comment 24 Sachin Ghai 2016-06-03 07:50:18 UTC
Created attachment 1164365 [details]
upgrade completed

Comment 25 Sachin Ghai 2016-06-03 08:30:07 UTC
Just to correct: Verification is done with upgrade from sat6.1.9 -> sat6.2 GA snap14.1 on rhel7. Rpm version is 14.2

satellite-6.2.0-14.2.el7sat.noarch

Comment 26 Bryan Kearney 2016-07-27 11:24:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501


Note You need to log in before you can comment on or make changes to this bug.