Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1398438 - Repository synchronization tries to create all the missing repositories on a remote capsule
Summary: Repository synchronization tries to create all the missing repositories on a ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: Brad Buckingham
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-24 21:39 UTC by Ivan Necas
Modified: 2020-01-17 16:15 UTC (History)
9 users (show)

Fixed In Version: rubygem-katello-3.0.0.90-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-19 08:18:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 17524 0 High Closed Repository synchronization tries to create all the missing repositories on a remote capsule 2020-09-02 15:21:05 UTC
Red Hat Product Errata RHBA-2016:2958 0 normal SHIPPED_LIVE Satellite 6.2.6 Async Bug Release 2016-12-19 13:17:24 UTC

Description Ivan Necas 2016-11-24 21:39:51 UTC 
Description of problem:

If library is synchronized into a capsule, the repository
synchornization also tries to get the content into this capsule.
However, as part of this effort, it checks for all repositories that
were not synchronized into a capsule, and tries to create them there.
This can have a big impact on performance, if there are a lot of
repositories in library, and a new capsule is added to the system to
be synchronized with the library.

Version-Release number of selected component (if applicable):
sat 6.2

How reproducible:
When library is synchonized to capsules

Steps to Reproduce:
1. have a large number of repositories in library
2. add new capsule to the system and add library as an environment to be synchronized
3. synchronize the repository

Actual results:
The synchronization of repository takes much longer, than it should,
the sync task in dynflow sits in "Actions::Katello::Repository::Sync" action,
a lot of trafic between sat and capsule due to checking for status.

Expected results: The sync task is not much slower than without a
capsule present. Only the specific repo is being created when missing on the capsule.
Comment 3 Ivan Necas 2016-11-28 17:46:19 UTC 
Also, this leads to a lot of 409 issues, when syncing several repos in the same organization at once, when the syncing is triggered

 RestClient::Conflict

409 Conflict

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
  `return!'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in
  `block in get_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
  `process_result'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in
  `block in transmit'"
- "/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in
  `transmit'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in
  `post'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in
  `get_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:22:in
  `create'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/extensions/repository.rb:71:in
  `create_with_importer_and_distributors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/pulp/repository/create.rb:27:in
Comment 4 Brad Buckingham 2016-11-29 20:27:21 UTC 
Based on discussion with Ivan, the following is the scenario causing issues:

1. Satellite: create/sync several (e.g. N) repositories to Library

2. Install a new capsule

3. Update the capsule include Library 
   (e.g. Infrastructure -> Capsules -> Edit (for the new capsule)

4. Sync a single repository
   (e.g. Content -> Sync Status)

As part of syncing that repository, observe that all N repositories will be created on the capsule [1] and only the desired repository will be synced [2].

In this scenario, the desired behavior would likely be to only create the repository that is being synced.

[1] Actions::Pulp::Repository::Create
[2] Actions::Pulp::Consumer::SyncCapsule
Comment 5 Brad Buckingham 2016-11-30 00:57:42 UTC 
Created redmine issue http://projects.theforeman.org/issues/17524 from this bug
Comment 6 Brad Buckingham 2016-11-30 21:15:57 UTC 
Upstream Katello PR: https://github.com/Katello/katello/pull/6475
Comment 7 Mike McCune 2016-12-01 22:55:03 UTC 
*** Hotfix Available ***

The below hotfix includes fixes for 3 bugs (including this bug) around content synchronization, memory consumption and performance.

This hotfix resolves:

BZ 1288656
BZ 1391704
BZ 1398438

Instructions for application:

1) download to your Satellite: http://people.redhat.com/~mmccune/hotfix/HOTFIX-1288656_1391704_1398438.tar.gz 

2) verify md5sum: 

dc877359bb61fb7c5d776889c7eec9c6  HOTFIX-1288656_1391704_1398438.tar.gz

3) extract and upgrade version in subdirectory depending on EL6 or EL7: 

rpm -Uvh tfm-rubygem-katello*.rpm

4) katello-service restart

5) resume operations
Comment 8 jcallaha 2016-12-13 15:31:41 UTC 
Verified in Satellite 6.2.6

Added a new capsule to the Satellite
Sync'd 1 of the 20+ repositories
Checked the content on the capsule
   Only the newly sync'd repo was added to the capsule

Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Execution type: Delayed
Start at: 2016-12-13 15:25:39 UTC
Start before: -
Started at: 2016-12-13 15:25:39 UTC
Ended at: 2016-12-13 15:25:59 UTC
State: stopped
Result: success
Params: repository '6 tools'; product '625'; organization 'Default Organization'


capsule filesystem
------------------
-bash-4.2# pwd
/var/lib/pulp/published/yum/https/repos/Default_Organization/Library

-bash-4.2# du -h
4.0K	./custom/625
8.0K	./custom
12K	.
Comment 10 errata-xmlrpc 2016-12-19 08:18:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2958
Note You need to log in before you can comment on or make changes to this bug.