Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1391704 - Repository synchronization checks all the repos on capsule on sync status
Summary: Repository synchronization checks all the repos on capsule on sync status
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks: 1353215 1394354
TreeView+ depends on / blocked
 
Reported: 2016-11-03 19:44 UTC by Ivan Necas
Modified: 2020-03-11 15:20 UTC (History)
18 users (show)

Fixed In Version: rubygem-katello-3.0.0.90-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-19 08:17:46 UTC
Target Upstream Version:
Embargoed:
cpatters: needinfo+

Attachments (Terms of Use)
tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm (4.85 MB, application/x-rpm)
2016-11-11 14:53 UTC, Zach Huntington-Meath 		no flags Details
tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm (102.33 KB, application/x-rpm)
2016-11-11 14:53 UTC, Zach Huntington-Meath 		no flags Details
tfm-rubygem-katello-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm (4.85 MB, application/x-rpm)
2016-11-18 15:20 UTC, Zach Huntington-Meath 		no flags Details
tfm-rubygem-katello_ostree-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm (102.30 KB, application/x-rpm)
2016-11-18 15:21 UTC, Zach Huntington-Meath 		no flags Details
Show Obsolete (2) View All


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 17211 0 Normal Closed Repository synchronization checks all the repos on capsule on sync status 2020-09-02 15:20:58 UTC
Red Hat Product Errata RHBA-2016:2958 0 normal SHIPPED_LIVE Satellite 6.2.6 Async Bug Release 2016-12-19 13:17:24 UTC

Description Ivan Necas 2016-11-03 19:44:45 UTC 
Description of problem: Part of the repo synchronizatoin is syncing
the rpms to capsules, if the capsule is syncing library. Due to a bug
in code, we are checking on sync status of all repositories in the
capsule (similar issue as described in
https://bugzilla.redhat.com/show_bug.cgi?id=1388296).

Version-Release number of selected component (if applicable):
sat 6.2

How reproducible:
When library is synchonized to capsules

Steps to Reproduce:
1. have a large number of repositories synced into capsule (100/1000+)
2. have a library in lifecycle environments synced to the capsule
3. synchronize the repository

Actual results:
The synchronization of repository takes much longer, than it should,
the sync task in dynflow sits in "Actions::Katello::Repository::Sync" action,
a lot of trafic between sat and capsule due to checking for status.

Expected results: The sync task is not much slower than without a
capsule present. Only check on the specific repo is happening between
the satellite and capsule.
Comment 1 Ivan Necas 2016-11-03 19:45:33 UTC 
Created redmine issue http://projects.theforeman.org/issues/17211 from this bug
Comment 2 Ivan Necas 2016-11-03 19:55:43 UTC 
Proposed patch https://github.com/Katello/katello/pull/6435
Comment 4 Bryan Kearney 2016-11-03 20:08:23 UTC 
Upstream bug assigned to inecas
Comment 5 Bryan Kearney 2016-11-03 20:08:25 UTC 
Upstream bug component is Performance
Comment 6 Bryan Kearney 2016-11-03 20:08:28 UTC 
Upstream bug assigned to inecas
Comment 8 Zach Huntington-Meath 2016-11-11 14:52:50 UTC 
HOTFIX INSTRUCTIONS

I've produced hotfix packages for the customer's installed version of Satellite (6.2.4 on el7) and tested on a similar Satellite.

Install instructions:

1) Download the attached RPMs, 

2) Stop katello-service

katello-service stop

3) Install package

yum localinstall tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm

4) Start katello-service

katello-service start

This should properly start Katello with the new changes in place.
Comment 9 Zach Huntington-Meath 2016-11-11 14:53:30 UTC 
Created attachment 1219817 [details]
tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm
Comment 10 Zach Huntington-Meath 2016-11-11 14:53:57 UTC 
Created attachment 1219818 [details]
tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm
Comment 11 Julio Entrena Perez 2016-11-11 15:02:16 UTC 
Thank you Zach.

If the _ostree package insn't installed in the Satellite, it's not necessary to install the _ostree hotfix package, correct?
Comment 12 Bryan Kearney 2016-11-11 15:08:06 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17211 has been resolved.
Comment 14 Julio Entrena Perez 2016-11-14 12:38:11 UTC 
I'm afraid that the hotfix triggered a problem:

A repository sync task fails on step "Actions::Katello::Repository::Sync" with error " ArgumentError, wrong number of arguments (2 for 1)".

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:791:in
  `distributors_match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:75:in
  `block in needs_distributor_updates'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:72:in
  `select'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:72:in
  `needs_distributor_updates'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb:83:in
  `repos_needing_updates'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb:23:in
  `plan'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/action.rb:461:in
  `block (3 levels) in execute_plan'"

Looking at /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb :

     71         def self.needs_distributor_updates(repos, capsule = nil)
     72           repos.select do |repo|
     73             repo_details = capsule ? capsule.pulp_repo_facts(repo.pulp_id) : repo.pulp_repo_facts
     74             next unless repo_details
     75             !repo.distributors_match?(repo_details["distributors"], capsule)
     76           end
     77         end
     78       end
     79     end

We call distributors_match?(repo_details["distributors"], capsule) (2 arguments) but it seems to accept only one?

    791       def distributors_match?(capsule_distributors)
    792         generated_distributor_configs = self.generate_distributors(true)
    793         generated_distributor_configs.all? do |gen_dist|
    794           type = gen_dist.class.type_id
    795           found_on_capsule = capsule_distributors.find { |dist| dist['distributor_type_id'] == type }
    796           found_on_capsule && filtered_distribution_config_equal?(gen_dist.config, found_on_capsule['config'])
    797         end
    798       end
Comment 16 Julio Entrena Perez 2016-11-14 13:59:27 UTC 
(In reply to Julio Entrena Perez from comment #14)

> A repository sync task fails on step "Actions::Katello::Repository::Sync"
> with error " ArgumentError, wrong number of arguments (2 for 1)".

This also happens when trying to publish a new version of any CV, even on CVs that don't have any custom repositories.
The error is displayed in the Publish New Version screen where you enter an optional comment and click Save to start publishing the new version (so the publishing of the new version does not even start).
Comment 18 Ivan Necas 2016-11-15 11:25:14 UTC 
I think there were issues conflict resolution when backporting the PR upstream to prepare a HOTFIX, I will work with Zach to get this resolved and provide a new hotfix
Comment 23 Zach Huntington-Meath 2016-11-18 15:20:36 UTC 
Created attachment 1221897 [details]
tfm-rubygem-katello-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm
Comment 24 Zach Huntington-Meath 2016-11-18 15:21:19 UTC 
Created attachment 1221898 [details]
tfm-rubygem-katello_ostree-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm
Comment 27 Zach Huntington-Meath 2016-11-22 16:47:15 UTC 
*** Bug 1388296 has been marked as a duplicate of this bug. ***
Comment 28 Zach Huntington-Meath 2016-11-22 16:47:53 UTC 
*** Bug 1394354 has been marked as a duplicate of this bug. ***
Comment 29 Ivan Necas 2016-11-24 21:15:27 UTC 
I've unmarked 1388296 and 1394354 as dupes of this, because I think the original issue with large environment syncing is there, while this bz (and corresponding patch) helps with repo syncing performance, that is influenced by the original issue, but this fix doesn't fix the original issue (which on the other hand doesn't happen that often as repo syncing)
Comment 38 jcallaha 2016-12-14 21:28:18 UTC 
Verified in Satellite 6.2.6 Snap 2

Have 324 total repositories sync'd to the capsule. No additional time seen during subsequent repository syncs. Typical sync is shown below. All other syncs performed with no detectable impact.

Id: 6d34b624-34dd-41a4-89ca-177bea2d53cf
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Execution type: Delayed
Start at: 2016-12-14 20:11:49 UTC
Start before: -
Started at: 2016-12-14 20:11:49 UTC
Ended at: 2016-12-14 20:12:10 UTC
State: stopped
Result: success
Params: repository '73 workstation'; product 'orion'; organization 'Default Organization'


Synchronize repository '6 capsule'; product '625'; organization 'Default Orga...	stopped	success	2016-12-14 20:58:53 UTC	2016-12-14 20:59:06 UTC	admin
Synchronize repository '7 tools'; product '625'; organization 'Default Organi...	stopped	success	2016-12-14 20:58:51 UTC	2016-12-14 20:59:02 UTC	admin
Synchronize repository '7 capsule'; product '625'; organization 'Default Orga...	stopped	success	2016-12-14 20:58:50 UTC	2016-12-14 20:59:02 UTC	admin
Synchronize repository '6 tools'; product '625'; organization 'Default Organi...	stopped	success	2016-12-14 20:58:53 UTC	2016-12-14 20:59:08 UTC	admin
Synchronize repository '6 satellite'; product '625'; organization 'Default Or...	stopped	success	2016-12-14 20:58:51 UTC	2016-12-14 20:59:06 UTC	admin
Synchronize repository '7 satellite'; product '625'; organization 'Default Or...	stopped	success	2016-12-14 20:58:50 UTC	2016-12-14 20:59:03 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:43 UTC	2016-12-14 21:09:07 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs i3...	stopped	success	2016-12-14 21:08:39 UTC	2016-12-14 21:09:06 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs i3...	stopped	success	2016-12-14 21:08:41 UTC	2016-12-14 21:09:01 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:40 UTC	2016-12-14 21:09:03 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:38 UTC	2016-12-14 21:09:07 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - Optional RPMs x86...	stopped	success	2016-12-14 21:08:43 UTC	2016-12-14 21:24:23 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - Supplementary RPM...	stopped	success	2016-12-14 21:08:45 UTC	2016-12-14 21:09:19 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - Extras RPMs x86_6...	stopped	success	2016-12-14 21:08:44 UTC	2016-12-14 21:09:06 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server...	stopped	success	2016-12-14 21:08:41 UTC	2016-12-14 21:13:49 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs ia...	stopped	success	2016-12-14 21:08:39 UTC	2016-12-14 21:08:53 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs ia...	stopped	success	2016-12-14 21:08:39 UTC	2016-12-14 21:08:53 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server - Optional RPMs x86...	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:19:53 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server...	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:26:47 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server Kickstart x86_64 7....	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:09:39 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:08:59 UTC	admin
Comment 40 errata-xmlrpc 2016-12-19 08:17:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2958
Note You need to log in before you can comment on or make changes to this bug.