Bug 1389802

Summary: upgrade to 6.2.3 failed at update_subscription_facet_backend_data (undefined method `inject' for nil:NilClass)
Product: Red Hat Satellite Reporter: Patrick C. F. Ernzer <pcfe>
Component: UpgradesAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Jitendra Yejare <jyejare>
Severity: medium Docs Contact:
Priority: high    
Version: 6.2.3CC: aperotti, bbuckingham, bkearney, cwelton, hmore, inecas, jsherril, lzap, sramacha, stbenjam, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.0.0.115-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1426397 (view as bug list) Environment:
Last Closed: 2017-05-01 13:56:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1399395, 1426397    
Attachments:
Description Flags
foreman-debug
none
LOG: foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data
none
shell output from an upgrade attempt none

Description Patrick C. F. Ernzer 2016-10-28 16:56:42 UTC
Created attachment 1215047 [details]
foreman-debug

Description of problem:
upgrading my test Sat 6.2.2 to 6.2.3 failed

failure was at the
  foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data
step


Version-Release number of selected component (if applicable):
6.2.3

How reproducible:
always (re-running "foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data" or "satellite-installer --scenario satellite --upgrade" does not help)

Steps to Reproduce:
1. upgrade 6.2.2 as per https://access.redhat.com/documentation/en/red-hat-satellite/6.2/single/installation-guide#updating_satellite_server_to_next_minor_version


Actual results:
tmp/subscription_facet_upgrade-1477666836.log
Error: sattestclient04.sattest.pcfe.net - 7

undefined method `inject' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/lib/katello/http_resource.rb:197:in `hash_to_query'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/lib/katello/resources/candlepin.rb:102:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/services/katello/candlepin/consumer.rb:17:in `block in <class:Consumer>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/lib/katello/lazy_accessor.rb:160:in `instance_eval'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/lib/katello/lazy_accessor.rb:160:in `run_initializer'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/lib/katello/lazy_accessor.rb:139:in `lazy_attribute_get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/app/lib/katello/lazy_accessor.rb:60:in `block (2 levels) in lazy_accessor'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake:43:in `block (5 levels) in <top (required)>'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/querying.rb:9:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.81/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake:41:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'


Expected results:
successful upgrade

Additional info:
post mess I looked at errata for the Sat itself and notived that it was not visible at https://access.redhat.com/management/distributors/ (maybe I searched badly)

subscription-manager register --type=satellite --force
subscription-manager attach --pool=SOMETHING
18:50:50 [root@satellite ~]# subscription-manager repos --list-enabled
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   rhel-server-rhscl-7-rpms
Repo Name: Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/os
Enabled:   1

Repo ID:   rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basearch/os
Enabled:   1

Repo ID:   rhel-7-server-satellite-6.2-rpms
Repo Name: Red Hat Satellite 6.2 (for RHEL 7 Server) (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/7Server/$basearch/satellite/6.2/os
Enabled:   1

18:51:07 [root@satellite ~]# 
does not help

Comment 1 Patrick C. F. Ernzer 2016-10-28 16:59:00 UTC
Created attachment 1215049 [details]
LOG: foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data

18:57:42 [root@satellite ~]#   foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data
Updating backend data for subscription facets
***********************************
*************WARNING***************
Errors detected during upgrade step.
Details saved to: /tmp/subscription_facet_upgrade-1477673881.log
This step can be rerun with:
  foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data
You are likely encountering a bug.
***********************************
18:58:01 [root@satellite ~]#

Comment 2 Patrick C. F. Ernzer 2016-10-28 17:00:39 UTC
Created attachment 1215050 [details]
shell output from an upgrade attempt

Comment 5 Patrick C. F. Ernzer 2016-10-28 17:39:38 UTC
note that the defined compute resource is the libvirt hypervisor the satellite itself runs as a VM on

19:08:06 [root@satellite ~]# hammer organization list
---|----------|----------|------------
ID | NAME     | LABEL    | DESCRIPTION
---|----------|----------|------------
1  | Sat Test | Sat_Test |   
---|----------|----------|------------
19:25:16 [root@satellite ~]# 
19:25:28 [root@satellite ~]# hammer organization create -name "Sat Test 2"
Could not create the organization:
  Error: Unrecognised option '-n'

  See: 'hammer organization create --help'
19:26:58 [root@satellite ~]# hammer organization create --name "Sat Test 2"
Organization created
19:27:07 [root@satellite ~]# hammer organization delete --id 1
Could not delete the organization:
  Failed to destroy a compute libvirt haswell (Libvirt) instance jenkins.sattest.pcfe.net: wrong number of arguments (0 for 1)
19:27:24 [root@satellite ~]# hammer compute-resource list
---|-----------------|---------
ID | NAME            | PROVIDER
---|-----------------|---------
1  | libvirt haswell | Libvirt
---|-----------------|---------
19:32:38 [root@satellite ~]# 
19:35:55 [root@satellite ~]# hammer compute-resource update --organization-ids 2 --id 1
Could not update the compute resource:
  Validation failed: Taxonomy has already been taken
19:36:26 [root@satellite ~]#

Comment 7 Harshad More 2016-12-01 19:02:41 UTC
One of the customer upgraded the satellite from 6.2.1 to 6.2.4 and faced similar error
Error was --
Upgrade completed, except for this error:

***********************************
*************WARNING***************
Errors detected during upgrade step.
Details saved to: /tmp/subscription_facet_upgrade-1480516196.log
This step can be rerun with:
  foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data
You are likely encountering a bug.
***********************************

Even after rerun facing the same error

Comment 10 Justin Sherrill 2016-12-08 15:06:36 UTC
Note that the error message:

Updating backend data for subscription facets
***********************************
*************WARNING***************
Errors detected during upgrade step.
Details saved to: /tmp/subscription_facet_upgrade-1481068894.log
This step can be rerun with:
  foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data
You are likely encountering a bug.
***********************************

is very generic and does not mean any specific bug.  Please do not associate any cases with this bug without verifying that the traceback in the log file matches the description from this bz.

Comment 11 Justin Sherrill 2016-12-08 15:09:47 UTC
The error "Katello::Resources::Candlepin::Consumer: 410 Gone" has nothing to do with the original report of this bz and will NOT be fixed as part of it.  See https://bugzilla.redhat.com/show_bug.cgi?id=1372720#c3 for more information.

This particular bug does seem like a simple case to workaround with a patch.  In the meantime you can resolve the issue with:

# formeman-rake console
> Katello::Host::SubscriptionFacet.where(:uuid => nil).destroy_all


I would like to reiterate, if you are not seeing the error 'undefined method inject of nil:NilClass', this will do NOTHING for you.

Comment 12 Justin Sherrill 2016-12-08 15:13:33 UTC
Created redmine issue http://projects.theforeman.org/issues/17612 from this bug

Comment 13 Bryan Kearney 2016-12-08 17:13:32 UTC
Upstream bug assigned to jsherril

Comment 14 Bryan Kearney 2016-12-08 17:13:38 UTC
Upstream bug assigned to jsherril

Comment 15 Satellite Program 2017-01-06 23:12:54 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17612 has been resolved.

Comment 16 Satellite Program 2017-02-23 21:10:00 UTC
Please add verifications steps for this bug to help QE verify

Comment 17 Justin Sherrill 2017-04-24 18:56:58 UTC
To reproduce:

1.  Register a client with subscription-manager
2.  Set its UUID to nil:

foreman-rake console:

>  ::Host.find_by(:name => "foo.example.com").subscription_facet.update_attributes!(:uuid => nil)

3. foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data

Comment 18 Jitendra Yejare 2017-04-25 14:59:28 UTC
Verified !

@ Satellite 6.2.9 snap 5


Steps:
1. Register a client with satellite.
2. Set its UUID to nil

irb(main):001:0> ::Host.find_by(:name => "0dockerclientrhel6").subscription_facet.update_attributes!(:uuid => nil)
=> true

3. foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data


Results:
The update subscription facet backed data command executed without any error stated in this bug description.

So moving this bug to verified state.

Comment 20 errata-xmlrpc 2017-05-01 13:56:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1191