Technically, the reason is twofold:

1) Adding two logs to /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/katello/util/cdn_var_substitutor.rb:

      def get_substitutions_from(base_path)
        Rails.logger.info("get_substitutions_from CDN: #{Time.now.strftime('%s.%N')} before get")
        ret = @resource.get(File.join(base_path, "listing")).split("\n")
        Rails.logger.info("get_substitutions_from CDN: #{Time.now.strftime('%s.%N')} after get")
        @good_listings << base_path
        ret
      rescue Errors::NotFound => e # some of listing file points to not existing content
        @bad_listings << base_path
        @resource.log :error, e.message
        [] # return no substitution for unreachable listings
      end

clearly shows the @resource.get call (i.e. contacting CDN over SSL and parsing output) consumes the most of the time - almost whole time between two consequtive "CDN: Requesting path .." calls. We can hardly improve this substantially.


2) The GET requests to CDN are called sequentially, due to "until paths_with_vars.empty?" cycle in the same file:

      def substitute_vars_in_prefix(prefix_with_vars)
        paths_with_vars = { {} => prefix_with_vars}
        prefixes_without_vars = @substitutions[prefix_with_vars]

        unless prefixes_without_vars
          prefixes_without_vars = {}
          until paths_with_vars.empty?
            substitutions, path = paths_with_vars.shift

            if substituable? path
              for_each_substitute_of_next_var substitutions, path do |new_substitution, new_path|
                begin
                  paths_with_vars[new_substitution] = new_path
                rescue Errors::SecurityViolation
                  # Some paths may not be accessible
                  @resource.log :warn, "#{new_path} is not accessible, ignoring"
                end
              end
            else
              prefixes_without_vars[substitutions] = path
            end
          end
          @substitutions[prefix_with_vars] = prefixes_without_vars
        end
        return prefixes_without_vars
      end


My suggestion is therefore to make the "until paths_with_vars.empty? { substitutions, path = paths_with_vars.shift .. } calls in parallel. Pseudocode (ignore my syntax errors):


until paths_with_vars.empty?
  for substitutions, paths in paths_with_vars do
    fork do
      (do all the stuff and append (new_substitution,new_path) key/value pair in a thread-safe way to new variable new_paths_with_vars instead of updating paths_with_vars)
    end
  end
  paths_with_vars = new_paths_with_vars
end


Eric, as github blames the code on you :) does the patch skeleton sound reasonable? Esp. forking bits?

If so I can try to propose a full patch.

Pavel, 

I think ehelms just moved the code around :)

I think its a valid strategy and I've implemented it with what looks like good results.  

-Justin

Created redmine issue http://projects.theforeman.org/issues/17564 from this bug

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17564 has been resolved.

I was able to expand several repository sets at the same time and most of them took less than three seconds. Good result!

Verified against Sat 6.3 Snap 9 => satellite-6.3.0-16.0.beta.el7sat.noarch

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336