Bug 1680441

Summary: customer db upgrade from 6.4 to 6.5 failed at upgrade task: katello:upgrades:3.11:update_puppet_repos
Product: Red Hat Satellite Reporter: Nikhil Kathole <nkathole>
Component: RepositoriesAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Nikhil Kathole <nkathole>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5.0CC: egolov, jsherril, mmccune, pcreech
Target Milestone: 6.5.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.10.0.22-1,tfm-rubygem-katello-3.10.0.23-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 12:40:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1653153    

Description Nikhil Kathole 2019-02-25 02:17:55 UTC
Description of problem:

Upgrade from 6.4 to 6.5 failed at upgrade task: katello:upgrades:3.11:update_puppet_repos


# foreman-rake katello:upgrades:3.11:update_puppet_repos --trace
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
** Invoke katello:upgrades:3.11:update_puppet_repos (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
** Execute katello:upgrades:3.11:update_puppet_repos
Refreshing repository R_CSL_4_Puppet (24)
Refreshing repository R_Puppet_Forge_Puppet (25)
Refreshing repository R_ALCATRAZ_Puppet (694)
Refreshing repository R_PKI_Puppet (27)
Refreshing repository R_DirectoryServices_Puppet (29)
Refreshing repository R_Storage_Puppet (31)
Refreshing repository R_Operations_Puppet (33)
Refreshing repository R_CSL_3_Puppet (39)
Refreshing repository R_OSMOSYS_Puppet (563)
Refreshing repository R_NetBackup_Puppet (44)
Refreshing repository R_PACEMAKER_Puppet (696)
Refreshing repository R_DOCKERMVP_Puppet (5560)
Refreshing repository R_OPENSHIFT_Puppet (2654)
Refreshing repository R_Capsule_Puppet (626)
Refreshing repository R_JAP9_Puppet (3670)
Refreshing repository R_TALOS_Puppet (9947)
Refreshing repository R_VIRTASSIST_Puppet (5738)
Refreshing repository R_JAP8_Puppet (1965)
Refreshing repository R_DHP-ORACLE_Puppet (1558)
Refreshing repository R_LinuxDesktop_Puppet (6360)
rake aborted!
ForemanTasks::TaskError: Task 55a0fdb5-c4ff-4b07-a79b-5f4e75a87646: RestClient::NotFound: 404 Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.4.3/lib/foreman_tasks.rb:55:in `block in sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.4.3/lib/foreman_tasks.rb:54:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.4.3/lib/foreman_tasks.rb:54:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/lib/katello/tasks/upgrades/3.11/update_puppet_repos.rake:14:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.21/lib/katello/tasks/upgrades/3.11/update_puppet_repos.rake:7:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:upgrades:3.11:update_puppet_repos


Version-Release number of selected component (if applicable):
Satellite 6.5 snap 17


How reproducible: always


Steps to Reproduce:
1. restore customer db
2. foreman-maintain upgrade run --whitelist="disk-performance" --target-version 6.5 -y

Actual results:

Upgrade failed at upgrade task: katello:upgrades:3.11:update_puppet_repos.


Expected results:
Sucessful upgrade.

Additional info:

# hammer task info --id 55a0fdb5-c4ff-4b07-a79b-5f4e75a87646
ID:          55a0fdb5-c4ff-4b07-a79b-5f4e75a87646
Action:      Refresh 
State:       paused
Result:      error
Started at:  2019/02/25 02:03:34
Ended at:    
Owner:       foreman_admin
Task errors: 404 Not Found

Comment 5 Evgeni Golov 2019-02-26 13:30:42 UTC
That's interesting, the traceback has RestClient::NotFound, but our rake task only has rescue RestClient::ResourceNotFound in it.

Justin, you had added this rescue in the first place, mind having a look?

Comment 6 Evgeni Golov 2019-02-26 13:34:54 UTC
seems those two should be aliases: https://github.com/rest-client/rest-client/blob/master/spec/unit/exceptions_spec.rb#L84-L86

Comment 7 Justin Sherrill 2019-02-26 13:38:12 UTC
yep!

> RestClient::ResourceNotFound == RestClient::NotFound
true


I think the problem is that this traceback is happening within the foreman-task, so foreman-task is throwing a NotFound, but its likely wrapped in its own exception.  

The strange thing here is that fetching the repository did NOT throw a NotFound, which means the repository exists, but the distributor its trying to update doesn't?   This are of code was changed recently so maybe there is some regression here (although i didn't think it would have been between these snaps).

Comment 8 Justin Sherrill 2019-02-26 16:40:05 UTC
Thinking about this a bit more, i highly suspect this is the cause:  https://projects.theforeman.org/issues/26005

and the fix has been merged upstream: https://github.com/Katello/katello/pull/7968

The root cause would have been introduced by https://github.com/Katello/katello/pull/7762, but probably wasn't involved in the upgrade process until https://bugzilla.redhat.com/show_bug.cgi?id=1665657 went in in snap 12.

However it is a timing issue, a very fast satellite may never see the issue, but a slower one probably would.

Comment 9 Mike McCune 2019-02-26 16:43:25 UTC
Is there a workaround if the upgrade fails? Can you just re-run it and likely get past it the 2nd time?

Comment 10 Justin Sherrill 2019-02-26 16:49:14 UTC
There really isn't a workaround, re-runs will likely hit the same issue again on slower hardware.

Comment 11 Justin Sherrill 2019-02-26 16:54:23 UTC
Connecting redmine issue https://projects.theforeman.org/issues/26005 from this bug

Comment 14 Nikhil Kathole 2019-02-28 13:44:12 UTC
VERIFIED

Version tested:
Satellite 6.5 snap 17.1

Comment 18 errata-xmlrpc 2019-05-14 12:40:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222