Bug 1398945
Summary: | Unpacking tree in Red Hat Repositories sequentially contacts CDN for each and every CDN path | ||
---|---|---|---|
Product: | Red Hat Satellite | Reporter: | Pavel Moravec <pmoravec> |
Component: | Repositories | Assignee: | Justin Sherrill <jsherril> |
Status: | CLOSED ERRATA | QA Contact: | Jonathon Turel <jturel> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 6.2.5 | CC: | bbuckingham, bkearney, ehelms, inecas, jcallaha, jsherril, jturel |
Target Milestone: | Unspecified | Keywords: | Performance, Triaged |
Target Release: | Unused | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-02-21 16:51:07 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Pavel Moravec
2016-11-27 14:00:16 UTC
Technically, the reason is twofold: 1) Adding two logs to /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/katello/util/cdn_var_substitutor.rb: def get_substitutions_from(base_path) Rails.logger.info("get_substitutions_from CDN: #{Time.now.strftime('%s.%N')} before get") ret = @resource.get(File.join(base_path, "listing")).split("\n") Rails.logger.info("get_substitutions_from CDN: #{Time.now.strftime('%s.%N')} after get") @good_listings << base_path ret rescue Errors::NotFound => e # some of listing file points to not existing content @bad_listings << base_path @resource.log :error, e.message [] # return no substitution for unreachable listings end clearly shows the @resource.get call (i.e. contacting CDN over SSL and parsing output) consumes the most of the time - almost whole time between two consequtive "CDN: Requesting path .." calls. We can hardly improve this substantially. 2) The GET requests to CDN are called sequentially, due to "until paths_with_vars.empty?" cycle in the same file: def substitute_vars_in_prefix(prefix_with_vars) paths_with_vars = { {} => prefix_with_vars} prefixes_without_vars = @substitutions[prefix_with_vars] unless prefixes_without_vars prefixes_without_vars = {} until paths_with_vars.empty? substitutions, path = paths_with_vars.shift if substituable? path for_each_substitute_of_next_var substitutions, path do |new_substitution, new_path| begin paths_with_vars[new_substitution] = new_path rescue Errors::SecurityViolation # Some paths may not be accessible @resource.log :warn, "#{new_path} is not accessible, ignoring" end end else prefixes_without_vars[substitutions] = path end end @substitutions[prefix_with_vars] = prefixes_without_vars end return prefixes_without_vars end My suggestion is therefore to make the "until paths_with_vars.empty? { substitutions, path = paths_with_vars.shift .. } calls in parallel. Pseudocode (ignore my syntax errors): until paths_with_vars.empty? for substitutions, paths in paths_with_vars do fork do (do all the stuff and append (new_substitution,new_path) key/value pair in a thread-safe way to new variable new_paths_with_vars instead of updating paths_with_vars) end end paths_with_vars = new_paths_with_vars end Eric, as github blames the code on you :) does the patch skeleton sound reasonable? Esp. forking bits? If so I can try to propose a full patch. Pavel, I think ehelms just moved the code around :) I think its a valid strategy and I've implemented it with what looks like good results. -Justin Created redmine issue http://projects.theforeman.org/issues/17564 from this bug Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17564 has been resolved. I was able to expand several repository sets at the same time and most of them took less than three seconds. Good result! Verified against Sat 6.3 Snap 9 => satellite-6.3.0-16.0.beta.el7sat.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
>
> For information on the advisory, and where to find the updated files, follow the link below.
>
> If the solution does not work for you, open a new bug report.
>
> https://access.redhat.com/errata/RHSA-2018:0336
|