Bug 1342014

Summary: Upgrade: POST "/api/hosts/facts" raises PG::Error: ERROR: relation "katello_content_facets" and "katello_subscription_facets" does not exist
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: UpgradesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, jsherril
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:09:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335807    
Attachments:
Description Flags
complete execption related to ERROR: relation "katello_content_facets" does not exist
none
complete exception related to ERROR: relation "katello_subscription_facets" does not exist
none
foreman-debug none

Description Sachin Ghai 2016-06-02 09:15:18 UTC
Description of problem:
I got following exception in production.log while performing upgrade from sat6.1.9 => sat6.2 GA snap14 on rhel7

==> /var/log/foreman/production.log <==
2016-06-02 03:18:28 [app] [I] Started POST "/api/hosts/facts" for 10.8.119.1 at 2016-06-02 03:18:28 -0400
2016-06-02 03:18:28 [app] [I] Processing by Api::V2::HostsController#facts as JSON
2016-06-02 03:18:28 [app] [I]   Parameters: {"facts"=>"[FILTERED]", "name"=>"qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com", "certname"=>"qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com", "apiv"=>"v2", :host=>{"name"=>"qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com", "certname"=>"qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com"}}
2016-06-02 03:18:28 [app] [I] Import facts for 'qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com' completed. Added: 11, Updated: 14, Deleted 2 facts
2016-06-02 03:18:28 [app] [W] Action failed
 | PG::Error: ERROR:  relation "katello_subscription_facets" does not exist
 | LINE 5:                WHERE a.attrelid = '"katello_subscription_fac...
 |                                           ^
 | 
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `block in exec_no_cache'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_adapter.rb:373:in `block in log'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_adapter.rb:367:in `log'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec_no_cache'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:137:in `exec_query'


==> /var/log/foreman/production.log <==
2016-06-02 03:18:29 [app] [I] Started POST "/api/reports" for 10.8.119.1 at 2016-06-02 03:18:29 -0400
2016-06-02 03:18:29 [app] [I] Processing by Api::V2::ReportsController#create as JSON
2016-06-02 03:18:29 [app] [I]   Parameters: {"report"=>"[FILTERED]", "apiv"=>"v2"}
2016-06-02 03:18:29 [app] [I] processing report for qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com
2016-06-02 03:18:29 [app] [W] Action failed
 | PG::Error: ERROR:  relation "katello_subscription_facets" does not exist
 | LINE 5:                WHERE a.attrelid = '"katello_subscription_fac...
 |                                           ^
 | 
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `block in exec_no_cache'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_adapter.rb:373:in `block in log'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract_adapter.rb:367:in `log'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `exec_no_cache'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:137:in `exec_query'


these execeptions were raised when installer failed at:

 Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [..................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.


Version-Release number of selected component (if applicable):
upgrade from sat6.1.9 => sat6.2 GA snap14 on rhel7

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
No exception should be raised in logs while performing upgrade.

Additional info:

Comment 1 Sachin Ghai 2016-06-02 09:17:00 UTC
Created attachment 1163990 [details]
complete execption related to ERROR:  relation "katello_content_facets" does not exist

Comment 2 Sachin Ghai 2016-06-02 09:18:24 UTC
Created attachment 1163991 [details]
complete exception related  to ERROR:  relation "katello_subscription_facets" does not exist

Comment 3 Sachin Ghai 2016-06-02 09:20:07 UTC
Created attachment 1164004 [details]
foreman-debug

Comment 5 Justin Sherrill 2016-06-02 12:29:43 UTC
The fact that this failed:

 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]

indicates that the database was not fully migrated and would be missing a bunch of tables, columns, and what not.  I would expect a TON of different errors when in this state.  It seems this was the result of https://bugzilla.redhat.com/show_bug.cgi?id=1341974

I would suggest this is NOTABUG as it should go away when https://bugzilla.redhat.com/show_bug.cgi?id=1341974 is fixed.  We can leave it open until if you'd like.

Comment 6 Sachin Ghai 2016-06-02 12:42:16 UTC
Thanks Justin, Lets keep it and will close it with next snap, if can't reproduce it.

Comment 7 Sachin Ghai 2016-06-03 07:57:41 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7.

I was not able to reproduce this issue. As mentioned in comment5, it was appearing due to a different bz 1341974. Moving this to verified.

And this time upgrade was completed successfully.

Comment 8 Sachin Ghai 2016-06-03 08:30:39 UTC
Just to correct: Verification is done with upgrade from sat6.1.9 -> sat6.2 GA snap14.1 on rhel7. Rpm version is 14.2

satellite-6.2.0-14.2.el7sat.noarch

Comment 9 Bryan Kearney 2016-07-27 11:09:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501