Bug 1665657

Summary:	Upgrade Step: katello:upgrades:3.11:update_puppet_repos failed during 6.4 to 6.5 upgrade
Product:	Red Hat Satellite	Reporter:	Nikhil Kathole <nkathole>
Component:	Repositories	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED ERRATA	QA Contact:	vijsingh
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.5.0	CC:	egolov, inecas, mbacovsk
Target Milestone:	6.5.0	Keywords:	Regression, Triaged, Upgrades
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-3.10.0.12-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-14 12:39:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1653153

Description Nikhil Kathole 2019-01-12 06:34:42 UTC

Description of problem:

Upgrade from 6.4 to 6.5 failed at Upgrade Step: katello:upgrades:3.11:update_puppet_repos

=============================================
Upgrade Step 1/7: katello:correct_repositories. This may take a long while.
Processing Repository 1/1: puppetrepo (1)
=============================================
Upgrade Step 2/7: katello:correct_puppet_environments. This may take a long while.
=============================================
Upgrade Step 3/7: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
0 orphaned consumer id(s) found in pulp.
Pulp orphaned consumers: []
=============================================
Upgrade Step 4/7: katello:upgrades:3.8:clear_checksum_type. =============================================
Upgrade Step 5/7: katello:upgrades:3.9:migrate_sync_plans. Starting recurring logic for migrated sync plans and deleting Pulp schedules
=============================================
Upgrade Step 6/7: katello:upgrades:3.10:clear_invalid_repo_credentials. =============================================
Upgrade Step 7/7: katello:upgrades:3.11:update_puppet_repos. Refreshing repository puppetrepo (1)
Failed upgrade task: katello:upgrades:3.11:update_puppet_repos, see logs for more information.
foreman-rake upgrade:run finished successfully!
Upgrade completed!


Version-Release number of selected component (if applicable): 
6.5.0 snap 11 


How reproducible:
always

Steps to Reproduce:
1. On satellite 6.4.1,

# hammer repository list
---|------------|-------------|--------------|--------------------------------------------
ID | NAME       | PRODUCT     | CONTENT TYPE | URL                                        
---|------------|-------------|--------------|--------------------------------------------
1  | puppetrepo | puppet repo | puppet       | http://omaciel.fedorapeople.org/bagoftricks
---|------------|-------------|--------------|--------------------------------------------
[root@qe-sat6-client-arch ~]# hammer repository info --id 1
ID:                 1
Name:               puppetrepo
Label:              puppetrepo
Organization:       Default Organization
Red Hat Repository: no
Content Type:       puppet
Mirror on Sync:     yes
URL:                http://omaciel.fedorapeople.org/bagoftricks
Publish Via HTTP:   yes
Published At:       http://qe-sat6-client-arch.usersys.redhat.com/pulp/puppet/7aa60922-5aaf-456a-9710-00ceb6b55bec/
Relative Path:      Default_Organization/Library/custom/puppet_repo/puppetrepo
Download Policy:    
Product:            
    ID:   1
    Name: puppet repo
GPG Key:            

Sync:               
    Status:         Success
    Last Sync Date: 39 minutes
Created:            2019/01/12 05:53:37
Updated:            2019/01/12 05:53:40
Content Counts:     
    Puppet Modules: 12


2. Enable 6.5 repos
3. foreman-maintain upgrade run --target-version 6.5


Actual results: Upgrade step  katello:upgrades:3.11:update_puppet_repos failed.


Expected results: Successful upgrade!


Additional info: 

# foreman-rake katello:upgrades:3.11:update_puppet_repos --trace
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
** Invoke katello:upgrades:3.11:update_puppet_repos (first_time)
** Invoke environment (first_time)
** Execute environment
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
** Invoke katello:upgrades:3.11:update_puppet_repos (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
Refreshing repository puppetrepo (1)
rake aborted!
NoMethodError: undefined method `[]' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/lib/katello/tasks/upgrades/3.11/update_puppet_repos.rake:10:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/lib/katello/tasks/upgrades/3.11/update_puppet_repos.rake:7:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:upgrades:3.11:update_puppet_repos
(See full trace by running task with --trace)
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.9/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
** Execute katello:upgrades:3.11:update_puppet_repos
Refreshing repository puppetrepo (1)
rake aborted!

Comment 5 Justin Sherrill 2019-01-15 14:20:20 UTC

Created redmine issue https://projects.theforeman.org/issues/25866 from this bug

Comment 6 Bryan Kearney 2019-01-15 15:05:01 UTC

Upstream bug assigned to jsherril

Comment 7 Bryan Kearney 2019-01-15 15:05:03 UTC

Upstream bug assigned to jsherril

Comment 8 Bryan Kearney 2019-01-15 15:05:49 UTC

Upstream bug assigned to jsherril

Comment 9 Bryan Kearney 2019-01-15 15:05:50 UTC

Upstream bug assigned to jsherril

Comment 10 Bryan Kearney 2019-01-15 15:06:41 UTC

Upstream bug assigned to jsherril

Comment 11 Bryan Kearney 2019-01-15 15:06:42 UTC

Upstream bug assigned to jsherril

Comment 12 Bryan Kearney 2019-01-15 15:07:52 UTC

Upstream bug assigned to jsherril

Comment 13 Bryan Kearney 2019-01-15 15:07:54 UTC

Upstream bug assigned to jsherril

Comment 14 Bryan Kearney 2019-01-15 15:08:46 UTC

Upstream bug assigned to jsherril

Comment 15 Bryan Kearney 2019-01-15 15:08:47 UTC

Upstream bug assigned to jsherril

Comment 16 Bryan Kearney 2019-01-17 15:04:56 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25866 has been resolved.

Comment 18 vijsingh 2019-01-22 07:21:08 UTC

Verified:

@Satellite 6.5.0 Snap 12.0

Automation Upgrade passed.

Comment 21 errata-xmlrpc 2019-05-14 12:39:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222