Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1607886 - Upgrade from 6.3 to 6.4 failed at Upgrade Step: set_upstream_pool_id
Summary: Upgrade from 6.3 to 6.4 failed at Upgrade Step: set_upstream_pool_id
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Subscription Management
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.4.0
Assignee: Partha Aji
QA Contact: Nikhil Kathole
URL:
Whiteboard:
Depends On: 1608426
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-24 13:36 UTC by Nikhil Kathole
Modified: 2019-11-05 23:30 UTC (History)
4 users (show)

Fixed In Version: tfm-rubygem-katello-3.7.0.9-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1608426 (view as bug list)
Environment:
Last Closed: 2018-10-16 19:26:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 24436 0 Normal Closed Upgrade from 6.3 to 6.4 failed at Upgrade Step: set_upstream_pool_id 2020-07-06 20:49:07 UTC

Description Nikhil Kathole 2018-07-24 13:36:53 UTC 
Description of problem: While upgrading satellite 6.3 to 6.4 snap 13, upgrade failed at 

Upgrade Step: set_upstream_pool_id (this may take a while) ...
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
rake aborted!
TypeError: no implicit conversion of String into Integer
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/pool.rb:167:in `[]'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/pool.rb:167:in `block in import_hosts'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/pool.rb:167:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/pool.rb:167:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/pool.rb:167:in `import_hosts'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/pool.rb:186:in `import_managed_associations'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/candlepin_object.rb:45:in `block (2 levels) in import_all'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `block in import_all'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/app/models/katello/glue/candlepin/candlepin_object.rb:38:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1.6/lib/katello/tasks/import_subscriptions.rake:6:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions
foreman-rake katello:import_subscriptions failed! Check the output for error!
Upgrade step set_upstream_pool_id failed. Check logs for more information


Version-Release number of selected component (if applicable):

Satellite 6.4 snap 13


How reproducible: always


Steps to Reproduce:
1. Restore 6.3 db
2. Add satellite 6.4 repos
3. satellite-installer --scenario satellite --upgrade

Actual results: Upgrade failed at upgrade Step: set_upstream_pool_id showing 
TypeError: no implicit conversion of String into Integer.

Expected results: Successful upgrade.


Additional info:
Comment 4 Justin Sherrill 2018-07-25 01:03:16 UTC 
Kevin and Bcourt... I need help with this. 

Katello is making a request to candlepin:

GET /candlepin/pools/29b8d2cd581a95a30158876f68e00c4f/entitlements?include="consumer.uuid"

And this request is taking a LONG time.  It seems like close to an hour.  After adding some debugging i discovered that candlepin is returning:

{"displayMessage"=>"OAuth problem encountered. Internal message is: timestamp_refused", "requestUuid"=>"5ab28817-cc3a-4826-957f-fd6af5608e95"}

Other pools seem to return with no issue.  I tried adding paging:

/candlepin/pools/29b8d2cd581a95a30158876f68e00c4f/entitlements?per_page=100&page=1

with the same result.  There are ~12,000 consumers attached to this pool, but watching 'top' i see postgresql take quite a bit of cpu for a very short time (just a second or so as expected), but then java takes 200-300% cpu for the full hour.
Comment 5 Barnaby Court 2018-07-26 02:15:32 UTC 
I'm removing the needinfo as BZ's have been opened upstream to track the work to either fix this in candlepin or provide an alternate API.
Comment 6 Justin Sherrill 2018-07-27 20:13:06 UTC 
Created redmine issue https://projects.theforeman.org/issues/24436 from this bug
Comment 9 Brad Buckingham 2018-07-31 15:01:11 UTC 
Moving the bugzilla to POST.  The candlepin changes have been completed and are in builds upstream.  The katello changes have been completed, tested and merged upstream.
Comment 11 Nikhil Kathole 2018-08-03 16:30:24 UTC 
VERIFIED

Version tested:
Satellite 6.4 snap 15

upgrade is successful with respective db.
Comment 13 Bryan Kearney 2018-10-16 19:26:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927
Note You need to log in before you can comment on or make changes to this bug.