Bug 1398438

Summary:	Repository synchronization tries to create all the missing repositories on a remote capsule
Product:	Red Hat Satellite	Reporter:	Ivan Necas <inecas>
Component:	Content Management	Assignee:	Brad Buckingham <bbuckingham>
Status:	CLOSED ERRATA	QA Contact:	jcallaha
Severity:	high	Docs Contact:
Priority:	high
Version:	6.2.0	CC:	aperotti, bbuckingham, bkearney, chrobert, daniele, jcallaha, mmccune, sthirugn, wpinheir
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-katello-3.0.0.90-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-19 08:18:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ivan Necas 2016-11-24 21:39:51 UTC

Description of problem:

If library is synchronized into a capsule, the repository
synchornization also tries to get the content into this capsule.
However, as part of this effort, it checks for all repositories that
were not synchronized into a capsule, and tries to create them there.
This can have a big impact on performance, if there are a lot of
repositories in library, and a new capsule is added to the system to
be synchronized with the library.

Version-Release number of selected component (if applicable):
sat 6.2

How reproducible:
When library is synchonized to capsules

Steps to Reproduce:
1. have a large number of repositories in library
2. add new capsule to the system and add library as an environment to be synchronized
3. synchronize the repository

Actual results:
The synchronization of repository takes much longer, than it should,
the sync task in dynflow sits in "Actions::Katello::Repository::Sync" action,
a lot of trafic between sat and capsule due to checking for status.

Expected results: The sync task is not much slower than without a
capsule present. Only the specific repo is being created when missing on the capsule.

Comment 3 Ivan Necas 2016-11-28 17:46:19 UTC

Also, this leads to a lot of 409 issues, when syncing several repos in the same organization at once, when the syncing is triggered

 RestClient::Conflict

409 Conflict

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
  `return!'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in
  `block in get_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in
  `process_result'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:50:in
  `block in transmit'"
- "/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in
  `transmit'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in
  `post'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in
  `get_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:22:in
  `create'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/extensions/repository.rb:71:in
  `create_with_importer_and_distributors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/actions/pulp/repository/create.rb:27:in

Comment 4 Brad Buckingham 2016-11-29 20:27:21 UTC

Based on discussion with Ivan, the following is the scenario causing issues:

1. Satellite: create/sync several (e.g. N) repositories to Library

2. Install a new capsule

3. Update the capsule include Library 
   (e.g. Infrastructure -> Capsules -> Edit (for the new capsule)

4. Sync a single repository
   (e.g. Content -> Sync Status)

As part of syncing that repository, observe that all N repositories will be created on the capsule [1] and only the desired repository will be synced [2].

In this scenario, the desired behavior would likely be to only create the repository that is being synced.

[1] Actions::Pulp::Repository::Create
[2] Actions::Pulp::Consumer::SyncCapsule

Comment 5 Brad Buckingham 2016-11-30 00:57:42 UTC

Created redmine issue http://projects.theforeman.org/issues/17524 from this bug

Comment 6 Brad Buckingham 2016-11-30 21:15:57 UTC

Upstream Katello PR: https://github.com/Katello/katello/pull/6475

Comment 7 Mike McCune 2016-12-01 22:55:03 UTC

*** Hotfix Available ***

The below hotfix includes fixes for 3 bugs (including this bug) around content synchronization, memory consumption and performance.

This hotfix resolves:

BZ 1288656
BZ 1391704
BZ 1398438

Instructions for application:

1) download to your Satellite: http://people.redhat.com/~mmccune/hotfix/HOTFIX-1288656_1391704_1398438.tar.gz 

2) verify md5sum: 

dc877359bb61fb7c5d776889c7eec9c6  HOTFIX-1288656_1391704_1398438.tar.gz

3) extract and upgrade version in subdirectory depending on EL6 or EL7: 

rpm -Uvh tfm-rubygem-katello*.rpm

4) katello-service restart

5) resume operations

Comment 8 jcallaha 2016-12-13 15:31:41 UTC

Verified in Satellite 6.2.6

Added a new capsule to the Satellite
Sync'd 1 of the 20+ repositories
Checked the content on the capsule
   Only the newly sync'd repo was added to the capsule

Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Execution type: Delayed
Start at: 2016-12-13 15:25:39 UTC
Start before: -
Started at: 2016-12-13 15:25:39 UTC
Ended at: 2016-12-13 15:25:59 UTC
State: stopped
Result: success
Params: repository '6 tools'; product '625'; organization 'Default Organization'


capsule filesystem
------------------
-bash-4.2# pwd
/var/lib/pulp/published/yum/https/repos/Default_Organization/Library

-bash-4.2# du -h
4.0K	./custom/625
8.0K	./custom
12K	.

Comment 10 errata-xmlrpc 2016-12-19 08:18:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2958