Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1430823 - Satellite6.2.8 upgrade fail with error : Upgrade Step: set_virt_who_on_pools (this may take a while) ...
Summary: Satellite6.2.8 upgrade fail with error : Upgrade Step: set_virt_who_on_pools ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.2.8
Hardware: All
OS: All
urgent
urgent
Target Milestone: Unspecified
Assignee: Tom McKay
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks: Sat6_Upgrades
TreeView+ depends on / blocked
 
Reported: 2017-03-09 16:15 UTC by Prashant Waghmare
Modified: 2020-09-10 10:18 UTC (History)
21 users (show)

Fixed In Version: rubygem-katello-3.0.0.134-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1446717 (view as bug list)
Environment:
Last Closed: 2017-06-20 17:23:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 18879 0 High Closed upgrade fail with error : Upgrade Step: set_virt_who_on_pools (this may take a while) ... 2020-04-01 02:39:46 UTC
Red Hat Product Errata RHBA-2017:1553 0 normal SHIPPED_LIVE Satellite 6.2.10 Async Bug Release 2017-06-20 21:19:07 UTC

Description Prashant Waghmare 2017-03-09 16:15:35 UTC
Description of problem:

Activation keys keeps loading in satellite6.2.7

So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 but getting below error while upgrade:
==========================
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
NoMethodError: undefined method `redhat?' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions

Upgrade step set_virt_who_on_pools failed. Check logs for more information.
====================================

Version-Release number of selected component (if applicable):

Red hat Satellite6.2.8

How reproducible:

Frequently

Steps to Reproduce:

1. Customer upgraded server from Satellite6.1.x to Satellite6.2.7, but after this activation keys page was kept loading endlessly. So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 

2.customer has external virt-who server. Stopped the virt-who server.

3. But, when running upgrade step, it got stuck with error as below.


Actual results:

Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
NoMethodError: undefined method `redhat?' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions

Upgrade step set_virt_who_on_pools failed. Check logs for more information.

Expected results:

Upgrade should work without any error.

Additional info:

Upgrade shows warning as below before throwing the error as below:

Upgrade Step: migrate_foreman...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
true

/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here

/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
false
========================

Following things are checked on satellite server when upgrade fail.

1) > rpm -qa tfm-rubygem-katello
tfm-rubygem-katello-3.0.0.105-1.el7sat.noarch

> rpm -qa satellite
satellite-6.2.8-4.0.el7sat.noarch

2) > curl -k https://$(hostname -f):9090/features
["discovery","dynflow","openscap","pulp","puppet","puppetca","ssh"]

3) > hammer ping
candlepin:
    Status:          ok
    Server Response: Duration: 51ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 38ms
pulp:
    Status:          ok
    Server Response: Duration: 87ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 697ms


4) >  su - postgres -c "psql -d foreman -c \"select * from foreman_tasks_tasks where state != 'stopped'\""
/etc/profile: line 88: HISTFILE: readonly variable
                  id                  |              type               |                    label                    |     started_at
      | ended_at |  state  | result  |             external_id              | parent_task_id |          start_at          | start_befo
re
--------------------------------------+---------------------------------+---------------------------------------------+---------------
------+----------+---------+---------+--------------------------------------+----------------+----------------------------+-----------
---
 e2476ddb-6ba9-435c-b11f-2f3148935e20 | ForemanTasks::Task::DynflowTask | Actions::Katello::EventQueue::Monitor       | 2017-03-08 16:
08:31 |          | running | pending | aa2a255d-2102-4bfd-8e37-b26464ce6d0e |                | 2017-03-08 16:08:31.521042 |
 19573cbe-e57b-4503-bbd9-934e9b0a1230 | ForemanTasks::Task::DynflowTask | Actions::Candlepin::ListenOnCandlepinEvents | 2017-03-08 16:
08:31 |          | running | pending | 4c1e8ea4-2cdb-46af-823c-8ce296430906 |                | 2017-03-08 16:08:31.419712 |
(2 rows)

5) tried reinstalling "tfm-rubygem-katello" but no luck.

Comment 1 dgupte 2017-03-09 20:55:27 UTC
Hi Team,

I am facing the similar issue during upgrade step in 6.2.8. 

Also, see "CandlepinResource: 404 Resource Not Found" in the error. Is this related?


[ INFO 2017-03-09 15:01:42 main] Upgrade Step: set_virt_who_on_pools (this may take a while) ...
[ERROR 2017-03-09 15:01:58 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
RestClient::ResourceNotFound: Katello::Resources::Candlepin::CandlepinResource: 404 Resource Not Found  (GET /candlepin/owners/Bloomberg_BNA/subscriptions)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/lib/katello/http_resource.rb:84:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/lib/katello/resources/candlepin.rb:576:in `get_for_owner'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/subscription.rb:14:in `get_for_owner'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:13:in `get_candlepin_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:17:in `import_candlepin_ids'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:38:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:37:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:7:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions

Comment 5 Tom McKay 2017-03-13 13:55:56 UTC
Created redmine issue http://projects.theforeman.org/issues/18879 from this bug

Comment 8 dgupte 2017-03-13 17:01:36 UTC
Hi Tom,

Okay, will file a different bz for it. 

Thanks

Comment 11 Satellite Program 2017-03-16 20:03:38 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18879 has been resolved.

Comment 15 Chris Roberts 2017-03-24 16:26:38 UTC
Created attachment 1266205 [details]
hotfix

Hotfix patch tarball

Usage (tarball extraction):

  tar -xzf satellite-hotfix-1430823.tar.gz
  cd satellite-hotfix-1430823

Usage (dry run):

  ./satellite-hotfix -n 1430823

Usage:

  ./satellite-hotfix 1430823

Revert a hotfix:

  ./satellite-hotfix -r 1430823

List all avaiable hotfixes:

  ./satellite-hotfix

The numbers are real Red Hat Bugzilla numbers. All backup and reject files are
save in backups/ sub-directory. Please keep the tarball for possible reverting
of the change in the future. If patching fails, revert immediately and report
the output. After each Satellite update, this hotfix process must be repeated
if the updated does not include the bugfix yet, search in errata notes for
bugzilla numbers to confirm.

If system-wide Ruby is not installed for some reason, use Satellite Foreman
Ruby software collection:

  scl enable rh-ruby22 -- ./satellite-hotfix -h

To find out Ruby versions available do:

  ls /opt/*/*ruby* -d

Comment 21 Dana Safford 2017-03-27 19:51:12 UTC
I set the Customer Escalation flag, as Sasktel is getting hotter.

Sasktel replied with evidence they have no  subscriptions with a null virt-who value. 

Can Sasktel run the current fix?

We will need a hotfix.

Comment 38 jcallaha 2017-06-08 16:05:02 UTC
Verified in Satellite 6.2.10 Snap 3

Setup virt-who on the 6.2.9 systems, then registered a few guests of a hypervisor. The upgrades to 6.2.10 went smoothly, with no issues during the set_virt_who_on_pools upgrade step.


-bash-4.2# time satellite-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
...
Upgrade Step: remove_event_queue...
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.43/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.137/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
Importing Subscriptions

Upgrade completed!

real	10m1.802s
user	6m50.519s
sys	0m39.406s

Comment 40 errata-xmlrpc 2017-06-20 17:23:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553


Note You need to log in before you can comment on or make changes to this bug.