Description of problem:
During a manifest refresh, all enabled (Red Hat) repositories are refreshed via Actions::Pulp3::Orchestration::Repository::RefreshIfNeeded dynflow step. That step is called sequentially for each such repo.

Since pulp can easily parallelize this independent work, we shall speed up the manifest refresh by calling the dynflow steps concurrently.

To prevent some scalability issues similarly like with Capsule sync, let add concurrency of blocks of :foreman_proxy_content_batch_size tasks "only".

Proposed patch (that I will raise upstream ticket and PR later on):

--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.57/app/lib/actions/katello/organization/manifest_refresh.rb.orig	2022-07-06 08:42:19.026663954 +0200
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.57/app/lib/actions/katello/organization/manifest_refresh.rb	2022-07-06 09:08:44.345836777 +0200
@@ -42,10 +42,14 @@ module Actions
 
         def plan_refresh_repos(import_products_action, org)
           repositories = ::Katello::Repository.in_default_view.in_product(::Katello::Product.redhat.in_org(org))
-          repositories.each do |repo|
-            plan_action(Katello::Repository::RefreshRepository,
-              repo,
-              :dependency => import_products_action.output)
+          repositories.in_groups_of(Setting[:foreman_proxy_content_batch_size], false) do |repo_batch|
+            concurrence do
+              repo_batch.each do |repo|
+                plan_action(Katello::Repository::RefreshRepository,
+                  repo,
+                  :dependency => import_products_action.output)
+              end
+            end
           end
         end
 

Having 60 Red Hat repos enabled, the patch improved time of manifest refresh from 100 seconds to approx 60. (while most fo the remaining time was spent in candlepin). So improvement in tens of percents of running time can be achieved in general (depending on # of enabled RH repos).


Version-Release number of selected component (if applicable):
Sat6.10


How reproducible:
100%


Steps to Reproduce:
1. Enable (dont necessarily sync) tens of Red Hat repositories
2. Refresh manifest a few times (to have statistically significant data)
3. Check its duration and sequence/concurrency of Actions::Pulp3::Orchestration::Repository::RefreshIfNeeded steps.
4. Apply the above patch and restart services
5. Run manifest refresh a few times again


Actual results:
3. Manifest refresh triggers RefreshIfNeeded steps in sequence.
5. Evident improvement in manifest refresh duration


Expected results:
3. to have the steps running concurrently (up to some limit, like the :foreman_proxy_content_batch_size tunable).



Additional info: