Bug 1391704

Summary: Repository synchronization checks all the repos on capsule on sync status
Product: Red Hat Satellite Reporter: Ivan Necas <inecas>
Component: Content ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: high    
Version: 6.2.0CC: abalakht, aperotti, bbuckingham, bkearney, cpatters, daniele, dcaplan, ggatward, inecas, jcallaha, jentrena, mmccune, pdwyer, sauchter, sthirugn, tspeetje, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: Performance, PrioBumpPM, Triaged
Target Release: UnusedFlags: cpatters: needinfo+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-katello-3.0.0.90-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-19 08:17:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1353215, 1394354    
Attachments:
Description Flags
tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm
none
tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm
none
tfm-rubygem-katello-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm
none
tfm-rubygem-katello_ostree-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm none

Description Ivan Necas 2016-11-03 19:44:45 UTC
Description of problem: Part of the repo synchronizatoin is syncing
the rpms to capsules, if the capsule is syncing library. Due to a bug
in code, we are checking on sync status of all repositories in the
capsule (similar issue as described in
https://bugzilla.redhat.com/show_bug.cgi?id=1388296).

Version-Release number of selected component (if applicable):
sat 6.2

How reproducible:
When library is synchonized to capsules

Steps to Reproduce:
1. have a large number of repositories synced into capsule (100/1000+)
2. have a library in lifecycle environments synced to the capsule
3. synchronize the repository

Actual results:
The synchronization of repository takes much longer, than it should,
the sync task in dynflow sits in "Actions::Katello::Repository::Sync" action,
a lot of trafic between sat and capsule due to checking for status.

Expected results: The sync task is not much slower than without a
capsule present. Only check on the specific repo is happening between
the satellite and capsule.

Comment 1 Ivan Necas 2016-11-03 19:45:33 UTC
Created redmine issue http://projects.theforeman.org/issues/17211 from this bug

Comment 2 Ivan Necas 2016-11-03 19:55:43 UTC
Proposed patch https://github.com/Katello/katello/pull/6435

Comment 4 Bryan Kearney 2016-11-03 20:08:23 UTC
Upstream bug assigned to inecas

Comment 5 Bryan Kearney 2016-11-03 20:08:25 UTC
Upstream bug component is Performance

Comment 6 Bryan Kearney 2016-11-03 20:08:28 UTC
Upstream bug assigned to inecas

Comment 8 Zach Huntington-Meath 2016-11-11 14:52:50 UTC
HOTFIX INSTRUCTIONS

I've produced hotfix packages for the customer's installed version of Satellite (6.2.4 on el7) and tested on a similar Satellite.

Install instructions:

1) Download the attached RPMs, 

2) Stop katello-service

katello-service stop

3) Install package

yum localinstall tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm

4) Start katello-service

katello-service start

This should properly start Katello with the new changes in place.

Comment 9 Zach Huntington-Meath 2016-11-11 14:53:30 UTC
Created attachment 1219817 [details]
tfm-rubygem-katello-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm

Comment 10 Zach Huntington-Meath 2016-11-11 14:53:57 UTC
Created attachment 1219818 [details]
tfm-rubygem-katello_ostree-3.0.0.82-2.HOTFIX_1288656_1391704.el7sat.noarch.rpm

Comment 11 Julio Entrena Perez 2016-11-11 15:02:16 UTC
Thank you Zach.

If the _ostree package insn't installed in the Satellite, it's not necessary to install the _ostree hotfix package, correct?

Comment 12 Bryan Kearney 2016-11-11 15:08:06 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17211 has been resolved.

Comment 14 Julio Entrena Perez 2016-11-14 12:38:11 UTC
I'm afraid that the hotfix triggered a problem:

A repository sync task fails on step "Actions::Katello::Repository::Sync" with error " ArgumentError, wrong number of arguments (2 for 1)".

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:791:in
  `distributors_match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:75:in
  `block in needs_distributor_updates'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:72:in
  `select'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/models/katello/glue/pulp/repo.rb:72:in
  `needs_distributor_updates'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb:83:in
  `repos_needing_updates'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb:23:in
  `plan'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/action.rb:461:in
  `block (3 levels) in execute_plan'"

Looking at /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/katello/capsule_content/sync.rb :

     71         def self.needs_distributor_updates(repos, capsule = nil)
     72           repos.select do |repo|
     73             repo_details = capsule ? capsule.pulp_repo_facts(repo.pulp_id) : repo.pulp_repo_facts
     74             next unless repo_details
     75             !repo.distributors_match?(repo_details["distributors"], capsule)
     76           end
     77         end
     78       end
     79     end

We call distributors_match?(repo_details["distributors"], capsule) (2 arguments) but it seems to accept only one?

    791       def distributors_match?(capsule_distributors)
    792         generated_distributor_configs = self.generate_distributors(true)
    793         generated_distributor_configs.all? do |gen_dist|
    794           type = gen_dist.class.type_id
    795           found_on_capsule = capsule_distributors.find { |dist| dist['distributor_type_id'] == type }
    796           found_on_capsule && filtered_distribution_config_equal?(gen_dist.config, found_on_capsule['config'])
    797         end
    798       end

Comment 16 Julio Entrena Perez 2016-11-14 13:59:27 UTC
(In reply to Julio Entrena Perez from comment #14)

> A repository sync task fails on step "Actions::Katello::Repository::Sync"
> with error " ArgumentError, wrong number of arguments (2 for 1)".

This also happens when trying to publish a new version of any CV, even on CVs that don't have any custom repositories.
The error is displayed in the Publish New Version screen where you enter an optional comment and click Save to start publishing the new version (so the publishing of the new version does not even start).

Comment 18 Ivan Necas 2016-11-15 11:25:14 UTC
I think there were issues conflict resolution when backporting the PR upstream to prepare a HOTFIX, I will work with Zach to get this resolved and provide a new hotfix

Comment 23 Zach Huntington-Meath 2016-11-18 15:20:36 UTC
Created attachment 1221897 [details]
tfm-rubygem-katello-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm

Comment 24 Zach Huntington-Meath 2016-11-18 15:21:19 UTC
Created attachment 1221898 [details]
tfm-rubygem-katello_ostree-3.0.0.82-2.bz1288656_1391704.el7sat.noarch.rpm

Comment 27 Zach Huntington-Meath 2016-11-22 16:47:15 UTC
*** Bug 1388296 has been marked as a duplicate of this bug. ***

Comment 28 Zach Huntington-Meath 2016-11-22 16:47:53 UTC
*** Bug 1394354 has been marked as a duplicate of this bug. ***

Comment 29 Ivan Necas 2016-11-24 21:15:27 UTC
I've unmarked 1388296 and 1394354 as dupes of this, because I think the original issue with large environment syncing is there, while this bz (and corresponding patch) helps with repo syncing performance, that is influenced by the original issue, but this fix doesn't fix the original issue (which on the other hand doesn't happen that often as repo syncing)

Comment 38 jcallaha 2016-12-14 21:28:18 UTC
Verified in Satellite 6.2.6 Snap 2

Have 324 total repositories sync'd to the capsule. No additional time seen during subsequent repository syncs. Typical sync is shown below. All other syncs performed with no detectable impact.

Id: 6d34b624-34dd-41a4-89ca-177bea2d53cf
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Execution type: Delayed
Start at: 2016-12-14 20:11:49 UTC
Start before: -
Started at: 2016-12-14 20:11:49 UTC
Ended at: 2016-12-14 20:12:10 UTC
State: stopped
Result: success
Params: repository '73 workstation'; product 'orion'; organization 'Default Organization'


Synchronize repository '6 capsule'; product '625'; organization 'Default Orga...	stopped	success	2016-12-14 20:58:53 UTC	2016-12-14 20:59:06 UTC	admin
Synchronize repository '7 tools'; product '625'; organization 'Default Organi...	stopped	success	2016-12-14 20:58:51 UTC	2016-12-14 20:59:02 UTC	admin
Synchronize repository '7 capsule'; product '625'; organization 'Default Orga...	stopped	success	2016-12-14 20:58:50 UTC	2016-12-14 20:59:02 UTC	admin
Synchronize repository '6 tools'; product '625'; organization 'Default Organi...	stopped	success	2016-12-14 20:58:53 UTC	2016-12-14 20:59:08 UTC	admin
Synchronize repository '6 satellite'; product '625'; organization 'Default Or...	stopped	success	2016-12-14 20:58:51 UTC	2016-12-14 20:59:06 UTC	admin
Synchronize repository '7 satellite'; product '625'; organization 'Default Or...	stopped	success	2016-12-14 20:58:50 UTC	2016-12-14 20:59:03 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:43 UTC	2016-12-14 21:09:07 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs i3...	stopped	success	2016-12-14 21:08:39 UTC	2016-12-14 21:09:06 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs i3...	stopped	success	2016-12-14 21:08:41 UTC	2016-12-14 21:09:01 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:40 UTC	2016-12-14 21:09:03 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:38 UTC	2016-12-14 21:09:07 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - Optional RPMs x86...	stopped	success	2016-12-14 21:08:43 UTC	2016-12-14 21:24:23 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - Supplementary RPM...	stopped	success	2016-12-14 21:08:45 UTC	2016-12-14 21:09:19 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server - Extras RPMs x86_6...	stopped	success	2016-12-14 21:08:44 UTC	2016-12-14 21:09:06 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server...	stopped	success	2016-12-14 21:08:41 UTC	2016-12-14 21:13:49 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs ia...	stopped	success	2016-12-14 21:08:39 UTC	2016-12-14 21:08:53 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 5 Server - RH Common RPMs ia...	stopped	success	2016-12-14 21:08:39 UTC	2016-12-14 21:08:53 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server - Optional RPMs x86...	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:19:53 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server...	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:26:47 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server Kickstart x86_64 7....	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:09:39 UTC	admin
Synchronize repository 'Red Hat Enterprise Linux 7 Server - RH Common RPMs x8...	stopped	success	2016-12-14 21:08:37 UTC	2016-12-14 21:08:59 UTC	admin

Comment 40 errata-xmlrpc 2016-12-19 08:17:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2958