Red Hat Bugzilla – Bug 1391704
Repository synchronization checks all the repos on capsule on sync status
Last modified: 2017-08-17 11:10:55 EDT
Description of problem: Part of the repo synchronizatoin is syncing the rpms to capsules, if the capsule is syncing library. Due to a bug in code, we are checking on sync status of all repositories in the capsule (similar issue as described in https://bugzilla.redhat.com/show_bug.cgi?id=1388296). Version-Release number of selected component (if applicable): sat 6.2 How reproducible: When library is synchonized to capsules Steps to Reproduce: 1. have a large number of repositories synced into capsule (100/1000+) 2. have a library in lifecycle environments synced to the capsule 3. synchronize the repository Actual results: The synchronization of repository takes much longer, than it should, the sync task in dynflow sits in "Actions::Katello::Repository::Sync" action, a lot of trafic between sat and capsule due to checking for status. Expected results: The sync task is not much slower than without a capsule present. Only check on the specific repo is happening between the satellite and capsule.
Created redmine issue http://projects.theforeman.org/issues/17211 from this bug
Proposed patch https://github.com/Katello/katello/pull/6435
Upstream bug assigned to inecas@redhat.com
Upstream bug component is Performance
HOTFIX INSTRUCTIONS I've produced hotfix packages for the customer's installed version of Satellite (6.2.4 on el7) and tested on a similar Satellite. Install instructions: 1) Download the attached RPMs, 2) Stop katello-service katello-service stop 3) Install package yum localinstall tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm 4) Start katello-service katello-service start This should properly start Katello with the new changes in place.
Created attachment 1219817 [details] tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm
Created attachment 1219818 [details] tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm
Thank you Zach. If the _ostree package insn't installed in the Satellite, it's not necessary to install the _ostree hotfix package, correct?
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17211 has been resolved.
I'm afraid that the hotfix triggered a problem: A repository sync task fails on step "Actions::Katello::Repository::Sync" with error " ArgumentError, wrong number of arguments (2 for 1)". --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:791:in `distributors_match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:75:in `block in needs_distributor_updates'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:72:in `select'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:72:in `needs_distributor_updates'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb:83:in `repos_needing_updates'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb:23:in `plan'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/action.rb:461:in `block (3 levels) in execute_plan'" Looking at /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb : 71 def self.needs_distributor_updates(repos, capsule = nil) 72 repos.select do |repo| 73 repo_details = capsule ? capsule.pulp_repo_facts(repo.pulp_id) : repo.pulp_repo_facts 74 next unless repo_details 75 !repo.distributors_match?(repo_details["distributors"], capsule) 76 end 77 end 78 end 79 end We call distributors_match?(repo_details["distributors"], capsule) (2 arguments) but it seems to accept only one? 791 def distributors_match?(capsule_distributors) 792 generated_distributor_configs = self.generate_distributors(true) 793 generated_distributor_configs.all? do |gen_dist| 794 type = gen_dist.class.type_id 795 found_on_capsule = capsule_distributors.find { |dist| dist['distributor_type_id'] == type } 796 found_on_capsule && filtered_distribution_config_equal?(gen_dist.config, found_on_capsule['config']) 797 end 798 end
(In reply to Julio Entrena Perez from comment #14) > A repository sync task fails on step "Actions::Katello::Repository::Sync" > with error " ArgumentError, wrong number of arguments (2 for 1)". This also happens when trying to publish a new version of any CV, even on CVs that don't have any custom repositories. The error is displayed in the Publish New Version screen where you enter an optional comment and click Save to start publishing the new version (so the publishing of the new version does not even start).
I think there were issues conflict resolution when backporting the PR upstream to prepare a HOTFIX, I will work with Zach to get this resolved and provide a new hotfix
Created attachment 1221897 [details] tfm-rubygem-katello-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm
Created attachment 1221898 [details] tfm-rubygem-katello_ostree-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm
*** Bug 1388296 has been marked as a duplicate of this bug. ***
*** Bug 1394354 has been marked as a duplicate of this bug. ***
I've unmarked 1388296 and 1394354 as dupes of this, because I think the original issue with large environment syncing is there, while this bz (and corresponding patch) helps with repo syncing performance, that is influenced by the original issue, but this fix doesn't fix the original issue (which on the other hand doesn't happen that often as repo syncing)
Verified in Satellite 6.2.6 Snap 2 Have 324 total repositories sync'd to the capsule. No additional time seen during subsequent repository syncs. Typical sync is shown below. All other syncs performed with no detectable impact. Id: 6d34b624-34dd-41a4-89ca-177bea2d53cf Label: Actions::Katello::Repository::Sync Name: Synchronize Owner: admin Execution type: Delayed Start at: 2016-12-14 20:11:49 UTC Start before: - Started at: 2016-12-14 20:11:49 UTC Ended at: 2016-12-14 20:12:10 UTC State: stopped Result: success Params: repository '73 workstation'; product 'orion'; organization 'Default Organization' Synchronize repository '6 capsule'; product '625'; organization 'Default Orga... stopped success 2016-12-14 20:58:53 UTC 2016-12-14 20:59:06 UTC admin Synchronize repository '7 tools'; product '625'; organization 'Default Organi... stopped success 2016-12-14 20:58:51 UTC 2016-12-14 20:59:02 UTC admin Synchronize repository '7 capsule'; product '625'; organization 'Default Orga... stopped success 2016-12-14 20:58:50 UTC 2016-12-14 20:59:02 UTC admin Synchronize repository '6 tools'; product '625'; organization 'Default Organi... stopped success 2016-12-14 20:58:53 UTC 2016-12-14 20:59:08 UTC admin Synchronize repository '6 satellite'; product '625'; organization 'Default Or... stopped success 2016-12-14 20:58:51 UTC 2016-12-14 20:59:06 UTC admin Synchronize repository '7 satellite'; product '625'; organization 'Default Or... stopped success 2016-12-14 20:58:50 UTC 2016-12-14 20:59:03 UTC admin Synchronize repository 'Red Hat Enterprise Linux 6 Server - RH Common RPMs x8... stopped success 2016-12-14 21:08:43 UTC 2016-12-14 21:09:07 UTC admin Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs i3... stopped success 2016-12-14 21:08:39 UTC 2016-12-14 21:09:06 UTC admin Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs i3... stopped success 2016-12-14 21:08:41 UTC 2016-12-14 21:09:01 UTC admin Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs x8... stopped success 2016-12-14 21:08:40 UTC 2016-12-14 21:09:03 UTC admin Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs x8... stopped success 2016-12-14 21:08:38 UTC 2016-12-14 21:09:07 UTC admin Synchronize repository 'Red Hat Enterprise Linux 6 Server - Optional RPMs x86... stopped success 2016-12-14 21:08:43 UTC 2016-12-14 21:24:23 UTC admin Synchronize repository 'Red Hat Enterprise Linux 6 Server - Supplementary RPM... stopped success 2016-12-14 21:08:45 UTC 2016-12-14 21:09:19 UTC admin Synchronize repository 'Red Hat Enterprise Linux 6 Server - Extras RPMs x86_6... stopped success 2016-12-14 21:08:44 UTC 2016-12-14 21:09:06 UTC admin Synchronize repository 'Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server... stopped success 2016-12-14 21:08:41 UTC 2016-12-14 21:13:49 UTC admin Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs ia... stopped success 2016-12-14 21:08:39 UTC 2016-12-14 21:08:53 UTC admin Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs ia... stopped success 2016-12-14 21:08:39 UTC 2016-12-14 21:08:53 UTC admin Synchronize repository 'Red Hat Enterprise Linux 7 Server - Optional RPMs x86... stopped success 2016-12-14 21:08:37 UTC 2016-12-14 21:19:53 UTC admin Synchronize repository 'Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server... stopped success 2016-12-14 21:08:37 UTC 2016-12-14 21:26:47 UTC admin Synchronize repository 'Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.... stopped success 2016-12-14 21:08:37 UTC 2016-12-14 21:09:39 UTC admin Synchronize repository 'Red Hat Enterprise Linux 7 Server - RH Common RPMs x8... stopped success 2016-12-14 21:08:37 UTC 2016-12-14 21:08:59 UTC admin
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:2958