Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours
Bug 1430823 - Satellite6.2.8 upgrade fail with error : Upgrade Step: set_virt_who_on_pools (this may take a while) ...
Satellite6.2.8 upgrade fail with error : Upgrade Step: set_virt_who_on_pools ...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades (Show other bugs)
6.2.8
All All
urgent Severity urgent (vote)
: 6.2.10
: Unused
Assigned To: Tom McKay
jcallaha
: Triaged
Depends On:
Blocks: Sat6_Upgrades
  Show dependency treegraph
 
Reported: 2017-03-09 11:15 EST by Prashant Waghmare
Modified: 2017-06-20 13:23 EDT (History)
21 users (show)

See Also:
Fixed In Version: rubygem-katello-3.0.0.134-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1446717 (view as bug list)
Environment:
Last Closed: 2017-06-20 13:23:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 18879 None None None 2017-03-13 09:56 EDT
Red Hat Product Errata RHBA-2017:1553 normal SHIPPED_LIVE Satellite 6.2.10 Async Bug Release 2017-06-20 17:19:07 EDT

  None (edit)
Description Prashant Waghmare 2017-03-09 11:15:35 EST
Description of problem:

Activation keys keeps loading in satellite6.2.7

So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 but getting below error while upgrade:
==========================
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
NoMethodError: undefined method `redhat?' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions

Upgrade step set_virt_who_on_pools failed. Check logs for more information.
====================================

Version-Release number of selected component (if applicable):

Red hat Satellite6.2.8

How reproducible:

Frequently

Steps to Reproduce:

1. Customer upgraded server from Satellite6.1.x to Satellite6.2.7, but after this activation keys page was kept loading endlessly. So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 

2.customer has external virt-who server. Stopped the virt-who server.

3. But, when running upgrade step, it got stuck with error as below.


Actual results:

Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
NoMethodError: undefined method `redhat?' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions

Upgrade step set_virt_who_on_pools failed. Check logs for more information.

Expected results:

Upgrade should work without any error.

Additional info:

Upgrade shows warning as below before throwing the error as below:

Upgrade Step: migrate_foreman...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
true

/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here

/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
false
========================

Following things are checked on satellite server when upgrade fail.

1) > rpm -qa tfm-rubygem-katello
tfm-rubygem-katello-3.0.0.105-1.el7sat.noarch

> rpm -qa satellite
satellite-6.2.8-4.0.el7sat.noarch

2) > curl -k https://$(hostname -f):9090/features
["discovery","dynflow","openscap","pulp","puppet","puppetca","ssh"]

3) > hammer ping
candlepin:
    Status:          ok
    Server Response: Duration: 51ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 38ms
pulp:
    Status:          ok
    Server Response: Duration: 87ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 697ms


4) >  su - postgres -c "psql -d foreman -c \"select * from foreman_tasks_tasks where state != 'stopped'\""
/etc/profile: line 88: HISTFILE: readonly variable
                  id                  |              type               |                    label                    |     started_at
      | ended_at |  state  | result  |             external_id              | parent_task_id |          start_at          | start_befo
re
--------------------------------------+---------------------------------+---------------------------------------------+---------------
------+----------+---------+---------+--------------------------------------+----------------+----------------------------+-----------
---
 e2476ddb-6ba9-435c-b11f-2f3148935e20 | ForemanTasks::Task::DynflowTask | Actions::Katello::EventQueue::Monitor       | 2017-03-08 16:
08:31 |          | running | pending | aa2a255d-2102-4bfd-8e37-b26464ce6d0e |                | 2017-03-08 16:08:31.521042 |
 19573cbe-e57b-4503-bbd9-934e9b0a1230 | ForemanTasks::Task::DynflowTask | Actions::Candlepin::ListenOnCandlepinEvents | 2017-03-08 16:
08:31 |          | running | pending | 4c1e8ea4-2cdb-46af-823c-8ce296430906 |                | 2017-03-08 16:08:31.419712 |
(2 rows)

5) tried reinstalling "tfm-rubygem-katello" but no luck.
Comment 1 dgupte 2017-03-09 15:55:27 EST
Hi Team,

I am facing the similar issue during upgrade step in 6.2.8. 

Also, see "CandlepinResource: 404 Resource Not Found" in the error. Is this related?


[ INFO 2017-03-09 15:01:42 main] Upgrade Step: set_virt_who_on_pools (this may take a while) ...
[ERROR 2017-03-09 15:01:58 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
RestClient::ResourceNotFound: Katello::Resources::Candlepin::CandlepinResource: 404 Resource Not Found  (GET /candlepin/owners/Bloomberg_BNA/subscriptions)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/lib/katello/http_resource.rb:84:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/lib/katello/resources/candlepin.rb:576:in `get_for_owner'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/subscription.rb:14:in `get_for_owner'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:13:in `get_candlepin_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:17:in `import_candlepin_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:38:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:37:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:7:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions
Comment 5 Tom McKay 2017-03-13 09:55:56 EDT
Created redmine issue http://projects.theforeman.org/issues/18879 from this bug
Comment 8 dgupte 2017-03-13 13:01:36 EDT
Hi Tom,

Okay, will file a different bz for it. 

Thanks
Comment 11 pm-sat@redhat.com 2017-03-16 16:03:38 EDT
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18879 has been resolved.
Comment 15 Chris Roberts 2017-03-24 12:26 EDT
Created attachment 1266205 [details]
hotfix

Hotfix patch tarball

Usage (tarball extraction):

  tar -xzf satellite-hotfix-1430823.tar.gz
  cd satellite-hotfix-1430823

Usage (dry run):

  ./satellite-hotfix -n 1430823

Usage:

  ./satellite-hotfix 1430823

Revert a hotfix:

  ./satellite-hotfix -r 1430823

List all avaiable hotfixes:

  ./satellite-hotfix

The numbers are real Red Hat Bugzilla numbers. All backup and reject files are
save in backups/ sub-directory. Please keep the tarball for possible reverting
of the change in the future. If patching fails, revert immediately and report
the output. After each Satellite update, this hotfix process must be repeated
if the updated does not include the bugfix yet, search in errata notes for
bugzilla numbers to confirm.

If system-wide Ruby is not installed for some reason, use Satellite Foreman
Ruby software collection:

  scl enable rh-ruby22 -- ./satellite-hotfix -h

To find out Ruby versions available do:

  ls /opt/*/*ruby* -d
Comment 21 Dana Safford 2017-03-27 15:51:12 EDT
I set the Customer Escalation flag, as Sasktel is getting hotter.

Sasktel replied with evidence they have no  subscriptions with a null virt-who value. 

Can Sasktel run the current fix?

We will need a hotfix.
Comment 38 jcallaha 2017-06-08 12:05:02 EDT
Verified in Satellite 6.2.10 Snap 3

Setup virt-who on the 6.2.9 systems, then registered a few guests of a hypervisor. The upgrades to 6.2.10 went smoothly, with no issues during the set_virt_who_on_pools upgrade step.


-bash-4.2# time satellite-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
...
Upgrade Step: remove_event_queue...
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.43/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.137/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
Importing Subscriptions

Upgrade completed!

real	10m1.802s
user	6m50.519s
sys	0m39.406s
Comment 40 errata-xmlrpc 2017-06-20 13:23:02 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553

Note You need to log in before you can comment on or make changes to this bug.