Description of problem: Activation keys keeps loading in satellite6.2.7 So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 but getting below error while upgrade: ========================== Upgrade Step: set_virt_who_on_pools (this may take a while) ... /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here rake aborted! NoMethodError: undefined method `redhat?' for nil:NilClass /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>' Tasks: TOP => katello:upgrades:3.3:import_subscriptions (See full trace by running task with --trace) Importing Subscriptions Upgrade step set_virt_who_on_pools failed. Check logs for more information. ==================================== Version-Release number of selected component (if applicable): Red hat Satellite6.2.8 How reproducible: Frequently Steps to Reproduce: 1. Customer upgraded server from Satellite6.1.x to Satellite6.2.7, but after this activation keys page was kept loading endlessly. So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 2.customer has external virt-who server. Stopped the virt-who server. 3. But, when running upgrade step, it got stuck with error as below. Actual results: Upgrade Step: set_virt_who_on_pools (this may take a while) ... /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here rake aborted! NoMethodError: undefined method `redhat?' for nil:NilClass /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>' Tasks: TOP => katello:upgrades:3.3:import_subscriptions (See full trace by running task with --trace) Importing Subscriptions Upgrade step set_virt_who_on_pools failed. Check logs for more information. Expected results: Upgrade should work without any error. Additional info: Upgrade shows warning as below before throwing the error as below: Upgrade Step: migrate_foreman... /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here true /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here false ======================== Following things are checked on satellite server when upgrade fail. 1) > rpm -qa tfm-rubygem-katello tfm-rubygem-katello-3.0.0.105-1.el7sat.noarch > rpm -qa satellite satellite-6.2.8-4.0.el7sat.noarch 2) > curl -k https://$(hostname -f):9090/features ["discovery","dynflow","openscap","pulp","puppet","puppetca","ssh"] 3) > hammer ping candlepin: Status: ok Server Response: Duration: 51ms candlepin_auth: Status: ok Server Response: Duration: 38ms pulp: Status: ok Server Response: Duration: 87ms foreman_tasks: Status: ok Server Response: Duration: 697ms 4) > su - postgres -c "psql -d foreman -c \"select * from foreman_tasks_tasks where state != 'stopped'\"" /etc/profile: line 88: HISTFILE: readonly variable id | type | label | started_at | ended_at | state | result | external_id | parent_task_id | start_at | start_befo re --------------------------------------+---------------------------------+---------------------------------------------+--------------- ------+----------+---------+---------+--------------------------------------+----------------+----------------------------+----------- --- e2476ddb-6ba9-435c-b11f-2f3148935e20 | ForemanTasks::Task::DynflowTask | Actions::Katello::EventQueue::Monitor | 2017-03-08 16: 08:31 | | running | pending | aa2a255d-2102-4bfd-8e37-b26464ce6d0e | | 2017-03-08 16:08:31.521042 | 19573cbe-e57b-4503-bbd9-934e9b0a1230 | ForemanTasks::Task::DynflowTask | Actions::Candlepin::ListenOnCandlepinEvents | 2017-03-08 16: 08:31 | | running | pending | 4c1e8ea4-2cdb-46af-823c-8ce296430906 | | 2017-03-08 16:08:31.419712 | (2 rows) 5) tried reinstalling "tfm-rubygem-katello" but no luck.
Hi Team, I am facing the similar issue during upgrade step in 6.2.8. Also, see "CandlepinResource: 404 Resource Not Found" in the error. Is this related? [ INFO 2017-03-09 15:01:42 main] Upgrade Step: set_virt_who_on_pools (this may take a while) ... [ERROR 2017-03-09 15:01:58 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here rake aborted! RestClient::ResourceNotFound: Katello::Resources::Candlepin::CandlepinResource: 404 Resource Not Found (GET /candlepin/owners/Bloomberg_BNA/subscriptions) /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result' /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in `block in transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in `get' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/lib/katello/http_resource.rb:84:in `get' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/lib/katello/resources/candlepin.rb:576:in `get_for_owner' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/subscription.rb:14:in `get_for_owner' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:13:in `get_candlepin_ids' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:17:in `import_candlepin_ids' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:38:in `block in import_all' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:37:in `import_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:7:in `block (4 levels) in <top (required)>' Tasks: TOP => katello:upgrades:3.3:import_subscriptions (See full trace by running task with --trace) Importing Subscriptions
Created redmine issue http://projects.theforeman.org/issues/18879 from this bug
Hi Tom, Okay, will file a different bz for it. Thanks
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18879 has been resolved.
Created attachment 1266205 [details] hotfix Hotfix patch tarball Usage (tarball extraction): tar -xzf satellite-hotfix-1430823.tar.gz cd satellite-hotfix-1430823 Usage (dry run): ./satellite-hotfix -n 1430823 Usage: ./satellite-hotfix 1430823 Revert a hotfix: ./satellite-hotfix -r 1430823 List all avaiable hotfixes: ./satellite-hotfix The numbers are real Red Hat Bugzilla numbers. All backup and reject files are save in backups/ sub-directory. Please keep the tarball for possible reverting of the change in the future. If patching fails, revert immediately and report the output. After each Satellite update, this hotfix process must be repeated if the updated does not include the bugfix yet, search in errata notes for bugzilla numbers to confirm. If system-wide Ruby is not installed for some reason, use Satellite Foreman Ruby software collection: scl enable rh-ruby22 -- ./satellite-hotfix -h To find out Ruby versions available do: ls /opt/*/*ruby* -d
I set the Customer Escalation flag, as Sasktel is getting hotter. Sasktel replied with evidence they have no subscriptions with a null virt-who value. Can Sasktel run the current fix? We will need a hotfix.
Verified in Satellite 6.2.10 Snap 3 Setup virt-who on the 6.2.9 systems, then registered a few guests of a hypervisor. The upgrades to 6.2.10 went smoothly, with no issues during the set_virt_who_on_pools upgrade step. -bash-4.2# time satellite-installer --upgrade Upgrading... Upgrade Step: stop_services... ... Upgrade Step: remove_event_queue... Upgrade Step: set_virt_who_on_pools (this may take a while) ... /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.43/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.137/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here Importing Subscriptions Upgrade completed! real 10m1.802s user 6m50.519s sys 0m39.406s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1553