Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1680441 - customer db upgrade from 6.4 to 6.5 failed at upgrade task: katello:upgrades:3.11:update_puppet_repos
Summary: customer db upgrade from 6.4 to 6.5 failed at upgrade task: katello:upgrades:...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.5.0
Assignee: Justin Sherrill
QA Contact: Nikhil Kathole
URL:
Whiteboard:
Depends On:
Blocks: 1653153
TreeView+ depends on / blocked
 
Reported: 2019-02-25 02:17 UTC by Nikhil Kathole
Modified: 2019-11-05 22:46 UTC (History)
4 users (show)

Fixed In Version: tfm-rubygem-katello-3.10.0.22-1,tfm-rubygem-katello-3.10.0.23-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:40:17 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 26005 0 None None None 2019-02-26 16:54:24 UTC
Foreman Issue Tracker 26171 0 None None None 2019-02-27 02:42:41 UTC
Red Hat Product Errata RHSA-2019:1222 0 None None None 2019-05-14 12:40:25 UTC

Description Nikhil Kathole 2019-02-25 02:17:55 UTC 
Description of problem:

Upgrade from 6.4 to 6.5 failed at upgrade task: katello:upgrades:3.11:update_puppet_repos


# foreman-rake katello:upgrades:3.11:update_puppet_repos --trace
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
** Invoke katello:upgrades:3.11:update_puppet_repos (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
** Execute katello:upgrades:3.11:update_puppet_repos
Refreshing repository R_CSL_4_Puppet (24)
Refreshing repository R_Puppet_Forge_Puppet (25)
Refreshing repository R_ALCATRAZ_Puppet (694)
Refreshing repository R_PKI_Puppet (27)
Refreshing repository R_DirectoryServices_Puppet (29)
Refreshing repository R_Storage_Puppet (31)
Refreshing repository R_Operations_Puppet (33)
Refreshing repository R_CSL_3_Puppet (39)
Refreshing repository R_OSMOSYS_Puppet (563)
Refreshing repository R_NetBackup_Puppet (44)
Refreshing repository R_PACEMAKER_Puppet (696)
Refreshing repository R_DOCKERMVP_Puppet (5560)
Refreshing repository R_OPENSHIFT_Puppet (2654)
Refreshing repository R_Capsule_Puppet (626)
Refreshing repository R_JAP9_Puppet (3670)
Refreshing repository R_TALOS_Puppet (9947)
Refreshing repository R_VIRTASSIST_Puppet (5738)
Refreshing repository R_JAP8_Puppet (1965)
Refreshing repository R_DHP-ORACLE_Puppet (1558)
Refreshing repository R_LinuxDesktop_Puppet (6360)
rake aborted!
ForemanTasks::TaskError: Task 55a0fdb5-c4ff-4b07-a79b-5f4e75a87646: RestClient::NotFound: 404 Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.4.3/lib/foreman_tasks.rb:55:in `block in sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.4.3/lib/foreman_tasks.rb:54:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.4.3/lib/foreman_tasks.rb:54:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/lib/katello/tasks/upgrades/3.11/update_puppet_repos.rake:14:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/lib/katello/tasks/upgrades/3.11/update_puppet_repos.rake:7:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:upgrades:3.11:update_puppet_repos


Version-Release number of selected component (if applicable):
Satellite 6.5 snap 17


How reproducible: always


Steps to Reproduce:
1. restore customer db
2. foreman-maintain upgrade run --whitelist="disk-performance" --target-version 6.5 -y

Actual results:

Upgrade failed at upgrade task: katello:upgrades:3.11:update_puppet_repos.


Expected results:
Sucessful upgrade.

Additional info:

# hammer task info --id 55a0fdb5-c4ff-4b07-a79b-5f4e75a87646
ID:          55a0fdb5-c4ff-4b07-a79b-5f4e75a87646
Action:      Refresh 
State:       paused
Result:      error
Started at:  2019/02/25 02:03:34
Ended at:    
Owner:       foreman_admin
Task errors: 404 Not Found
Comment 5 Evgeni Golov 2019-02-26 13:30:42 UTC 
That's interesting, the traceback has RestClient::NotFound, but our rake task only has rescue RestClient::ResourceNotFound in it.

Justin, you had added this rescue in the first place, mind having a look?
Comment 6 Evgeni Golov 2019-02-26 13:34:54 UTC 
seems those two should be aliases: https://github.com/rest-client/rest-client/blob/master/spec/unit/exceptions_spec.rb#L84-L86
Comment 7 Justin Sherrill 2019-02-26 13:38:12 UTC 
yep!

> RestClient::ResourceNotFound == RestClient::NotFound
true


I think the problem is that this traceback is happening within the foreman-task, so foreman-task is throwing a NotFound, but its likely wrapped in its own exception.  

The strange thing here is that fetching the repository did NOT throw a NotFound, which means the repository exists, but the distributor its trying to update doesn't?   This are of code was changed recently so maybe there is some regression here (although i didn't think it would have been between these snaps).
Comment 8 Justin Sherrill 2019-02-26 16:40:05 UTC 
Thinking about this a bit more, i highly suspect this is the cause:  https://projects.theforeman.org/issues/26005

and the fix has been merged upstream: https://github.com/Katello/katello/pull/7968

The root cause would have been introduced by https://github.com/Katello/katello/pull/7762, but probably wasn't involved in the upgrade process until https://bugzilla.redhat.com/show_bug.cgi?id=1665657 went in in snap 12.

However it is a timing issue, a very fast satellite may never see the issue, but a slower one probably would.
Comment 9 Mike McCune 2019-02-26 16:43:25 UTC 
Is there a workaround if the upgrade fails? Can you just re-run it and likely get past it the 2nd time?
Comment 10 Justin Sherrill 2019-02-26 16:49:14 UTC 
There really isn't a workaround, re-runs will likely hit the same issue again on slower hardware.
Comment 11 Justin Sherrill 2019-02-26 16:54:23 UTC 
Connecting redmine issue https://projects.theforeman.org/issues/26005 from this bug
Comment 14 Nikhil Kathole 2019-02-28 13:44:12 UTC 
VERIFIED

Version tested:
Satellite 6.5 snap 17.1
Comment 18 errata-xmlrpc 2019-05-14 12:40:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222
Note You need to log in before you can comment on or make changes to this bug.