Bug 1683687 - improve Katello::Pool.import_all by querying candlepin activation keys once per each org only
Summary: improve Katello::Pool.import_all by querying candlepin activation keys once p...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Activation Keys
Version: 6.4
Hardware: x86_64
OS: Linux
urgent
high vote
Target Milestone: Released
Assignee: Partha Aji
QA Contact: Perry Gagne
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-27 15:00 UTC by Pavel Moravec
Modified: 2019-10-07 17:18 UTC (History)
4 users (show)

Fixed In Version: tfm-rubygem-katello-3.10.0.30-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1717194 (view as bug list)
Environment:
Last Closed: 2019-05-14 12:40:17 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:40:25 UTC
Foreman Issue Tracker 26365 None None None 2019-03-14 18:21:13 UTC

Description Pavel Moravec 2019-02-27 15:00:47 UTC
Description of problem:
On a scaled environment with thousands of pools, katello:import_subscriptions (called e.g. during an upgrade) does call

Katello::Pool.import_all

that traverses over each and every pool and imports hosts and namely Activation Keys from candlepin. That means, a call like:

/candlepin/owners/RedHat/activation_keys/?include=id&include=pools.pool.id

is repeatedly requested for each and every pool.

If such a request takes longer, overall upgrade time is redundantly increased.

I propose a simple patch to query that once and pass the keys to the proper routine.


Version-Release number of selected component (if applicable):
Sat 6.4.2


How reproducible:
100%


Steps to Reproduce:
1. Have a system with some AKs and more pools in manifest - the more the better.

2. in one terminal:
tail -f /var/log/candlepin/candlepin.log | grep "/activation_keys/?include=id&include=pools.pool.id"

3. in 2nd terminal:
foreman-rake katello:import_subscriptions

4. count number of lines printed in the 1st terminal


Actual results:
4. shows as many logs / queries to candlepin as the number of pools is (plus one, I think).


Expected results:
4. to show just 1 or 2 such queries



Additional info:
Patch:

--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.46/app/models/katello/glue/candlepin/candlepin_object.rb.orig	2019-02-27 15:43:28.390609472 +0100
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.46/app/models/katello/glue/candlepin/candlepin_object.rb.new	2019-02-27 15:44:07.067642277 +0100
@@ -37,12 +37,13 @@ module Katello
 
         organizations.each do |org|
           candlepin_ids = import_candlepin_ids(org)
+          keys = Resources::Candlepin::ActivationKey.get(nil, "?include=id&include=pools.pool.id", org.label)
 
           objects = self.in_organization(org)
           objects.each do |item|
             if candlepin_ids.include?(item.cp_id)
               item.import_data
-              item.import_managed_associations if import_managed_associations && item.respond_to?(:import_managed_associations)
+              item.import_managed_associations(keys) if import_managed_associations && item.respond_to?(:import_managed_associations)
             else
               item.destroy
             end

--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.46/app/models/katello/glue/candlepin/pool.rb.orig	2019-02-27 15:44:38.758669157 +0100
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.46/app/models/katello/glue/candlepin/pool.rb.new 	2019-02-27 15:50:58.411991079 +0100
@@ -181,17 +181,16 @@ module Katello
         Katello::SubscriptionFacetPool.where(:pool_id => self.id, :subscription_facet_id => entries_to_remove).delete_all
       end
 
-      def import_managed_associations
+      def import_managed_associations(keys)
         import_hosts
-        create_activation_key_associations
+        create_activation_key_associations(keys)
       end
 
       def hosts
         ::Host.where(:id => self.subscription_facets.pluck(:host_id))
       end
 
-      def create_activation_key_associations
-        keys = Resources::Candlepin::ActivationKey.get(nil, "?include=id&include=pools.pool.id", organization.label)
+      def create_activation_key_associations(keys)
         activation_key_ids = keys.collect do |key|
           key['id'] if key['pools'].present? && key['pools'].any? { |pool| pool['pool'].try(:[], 'id') == cp_id }
         end

Comment 3 Pavel Moravec 2019-02-27 15:06:07 UTC
Re number of such queries after the patch: there should be as many as the number of organizations is (plus one), not "just" 1-2. But still significantly less than # of pools.

Comment 6 Partha Aji 2019-03-14 18:21:12 UTC
Connecting redmine issue https://projects.theforeman.org/issues/26365 from this bug

Comment 7 Bryan Kearney 2019-03-16 02:07:52 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/26365 has been resolved.

Comment 12 errata-xmlrpc 2019-05-14 12:40:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.