Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1399294 - Synchronizing a repository with large amount rpms causes large memory usages while indexing
Summary: Synchronizing a repository with large amount rpms causes large memory usages ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks: 1353215 1394313
TreeView+ depends on / blocked
 
Reported: 2016-11-28 17:13 UTC by Ivan Necas
Modified: 2021-12-10 14:48 UTC (History)
16 users (show)

Fixed In Version: rubygem-katello-3.0.0.90-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-19 08:18:09 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 17512 0 Normal Closed Synchronizing a repository with large amount rpms causes large memory usages while indexing 2020-09-02 15:21:01 UTC
Red Hat Knowledge Base (Solution) 2800381 0 None None None 2016-12-08 13:57:14 UTC
Red Hat Product Errata RHBA-2016:2958 0 normal SHIPPED_LIVE Satellite 6.2.6 Async Bug Release 2016-12-19 13:17:24 UTC

Description Ivan Necas 2016-11-28 17:13:05 UTC 
Description of problem:
When synchronizing repositories with large amount of RPMs causes the dynflow_executor process to grow, probably due to loading too much memory into variable while performing the indexing

Version-Release number of selected component (if applicable):
Sat 6.2

How reproducible:


Steps to Reproduce:
1. enable several large repositories (such as Red_Hat_Enterprise_Linux_Desktop-Red_Hat_Enterprise_Linux_5_Desktop_RPMs_x86_64_5Client, Red_Hat_Enterprise_Linux_Desktop-Red_Hat_Enterprise_Linux_5_Desktop_RPMs_x86_64_5_11, Red_Hat_Enterprise_Linux_Desktop-Red_Hat_Enterprise_Linux_5_Desktop_RPMs_i386_5_11, Red_Hat_Enterprise_Linux_Desktop-Red_Hat_Enterprise_Linux_5_Desktop_RPMs_i386_5Client)
2. synchronize the repositories (one repo should be enough, but the more running at once, the more the growth should be visible)
3. watch the RSS of dynflow_executor grow

Actual results:
The memory of the executor grows very vast

Expected results:
Much lower memory growth
Comment 3 Ivan Necas 2016-11-28 17:24:24 UTC 
The problem seems to be in the fact, that we accumulate all the metadata
when indexing the RPMS, and then perform the indexing, while we should do the indexing in slices, so that garbage collection can take it's part in the meantime,
see:

 https://github.com/Katello/katello/blob/ddf031d62943c122f0a877c4640686c1bae0c877/app/models/katello/glue/pulp/repo.rb#L403
 https://github.com/Katello/katello/blob/ddf031d62943c122f0a877c4640686c1bae0c877/app/models/katello/glue/pulp/repo.rb#L483
https://github.com/Katello/katello/blob/ddf031d62943c122f0a877c4640686c1bae0c877/app/models/katello/glue/pulp/repo.rb#L458

I have not performed deeper analysis of the code, this are just two obvious examples.
Comment 4 Justin Sherrill 2016-11-29 02:17:30 UTC 
Created redmine issue http://projects.theforeman.org/issues/17512 from this bug
Comment 9 Bryan Kearney 2016-12-07 17:09:45 UTC 
Upstream bug component is Repositories
Comment 11 Bryan Kearney 2016-12-08 03:09:39 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/17512 has been resolved.
Comment 12 Mike McCune 2016-12-12 20:45:29 UTC 
*** Hotfix Available ***

The below hotfix includes fixes for 4 bugs (including this bug) around content synchronization, memory consumption and performance.

This hotfix resolves:

BZ 1288656
BZ 1391704
BZ 1398438
BZ 1399294

Instructions for application:

1) download to your Satellite: http://people.redhat.com/~mmccune/hotfix/HOTFIX-BZ-1288656-1391704-1398438-1399294.tar.gz
2) verify md5sum: 

$ md5sum HOTFIX-BZ-1288656-1391704-1398438-1399294.tar.gz
6a47aa576cb18beb7bda19dec9149da8  HOTFIX-BZ-1288656-1391704-1398438-1399294.tar.gz

3) extract and upgrade version in subdirectory depending on EL6 or EL7: 

rpm -Uvh tfm-rubygem-katello*.rpm

4) katello-service restart

5) resume operations


The bugs mentioned in this comment are undergoing formalized testing to profile memory use but we wanted to make this available early for anyone who is interested in trying the fix before it is released as part of a 6.2.Z release.
Comment 13 Mike McCune 2016-12-12 20:56:45 UTC 
WARNING: We just released Satellite 6.2.5: 

https://access.redhat.com/errata/RHBA-2016:2940 

The hotfix in the previous comment is built against 6.2.4 and the fix for this bug and others mentioned are to be included in 6.2.6. 

Will attach a second hotfix for those using 6.2.5.
Comment 14 Mike McCune 2016-12-12 22:03:41 UTC 
*** HOTFIX For 6.2.5 ***

If you wish to apply the hotfix for this bug against 6.2.5, we have made it available here: 

http://people.redhat.com/~mmccune/hotfix/HOTFIX-6.2.5-BZ-1288656-1391704-1398438-1399294.tar.gz

Same instructions as Comment 12 but use the above tar.gz

$ md5sum HOTFIX-6.2.5-BZ-1288656-1391704-1398438-1399294.tar.gz
bb4c1135306fd3306a97b5cf30405947  HOTFIX-6.2.5-BZ-1288656-1391704-1398438-1399294.tar.gz
Comment 15 jcallaha 2016-12-13 16:45:48 UTC 
Verified in Satellite 6.2.6

Much smaller RSS growth seen during sync of these 4 repositories:
Red Hat Enterprise Linux 5 Desktop RPMs x86_64 5Client
Red Hat Enterprise Linux 5 Desktop RPMs x86_64 5.11
Red Hat Enterprise Linux 5 Desktop RPMs i386 5Client
Red Hat Enterprise Linux 5 Desktop RPMs i386 5.11


foreman   2266 11.0  0.4 2024728 312760 ?      Sl   10:55   0:20 dynflow_executor
---- Start Syncs ----
foreman   2266 11.0  0.4 2024728 313008 ?      Sl   10:55   0:21 dynflow_executor
foreman   2266 10.7  0.4 2091292 321284 ?      Sl   10:55   0:22 dynflow_executor
foreman   2266  9.9  0.4 2091292 323772 ?      Sl   10:55   0:23 dynflow_executor
foreman   2266  9.1  0.5 2224420 330184 ?      Sl   10:55   0:24 dynflow_executor
foreman   2266  8.3  0.5 2224420 333828 ?      Sl   10:55   0:25 dynflow_executor
foreman   2266  7.5  0.5 2224420 340096 ?      Sl   10:55   0:26 dynflow_executor
foreman   2266  7.2  0.5 2224420 341172 ?      Sl   10:55   0:27 dynflow_executor
foreman   2266  6.7  0.5 2224420 342028 ?      Sl   10:55   0:28 dynflow_executor
foreman   2266  6.3  0.5 2224420 342452 ?      Sl   10:55   0:29 dynflow_executor
foreman   2266  6.1  0.5 2224420 342704 ?      Sl   10:55   0:30 dynflow_executor
foreman   2266  5.9  0.5 2224420 343144 ?      Sl   10:55   0:31 dynflow_executor
foreman   2266  5.7  0.5 2224420 343572 ?      Sl   10:55   0:32 dynflow_executor
foreman   2266  5.5  0.5 2224420 344016 ?      Sl   10:55   0:33 dynflow_executor
foreman   2266  5.3  0.5 2224420 347244 ?      Sl   10:55   0:34 dynflow_executor
foreman   2266  5.2  0.5 2224420 347252 ?      Sl   10:55   0:35 dynflow_executor
foreman   2266  5.1  0.5 2224420 347308 ?      Sl   10:55   0:36 dynflow_executor
foreman   2266  4.9  0.5 2224420 347328 ?      Sl   10:55   0:37 dynflow_executor
foreman   2266  4.8  0.5 2224420 350096 ?      Sl   10:55   0:38 dynflow_executor
foreman   2266  4.7  0.5 2224420 351288 ?      Sl   10:55   0:39 dynflow_executor
foreman   2266  4.6  0.5 2224420 351936 ?      Sl   10:55   0:40 dynflow_executor
foreman   2266  4.6  0.5 2224420 353436 ?      Sl   10:55   0:41 dynflow_executor
foreman   2266  4.5  0.5 2224420 356160 ?      Sl   10:55   0:42 dynflow_executor
foreman   2266  4.4  0.5 2224420 359080 ?      Sl   10:55   0:43 dynflow_executor
foreman   2266  4.3  0.5 2224420 360604 ?      Sl   10:55   0:44 dynflow_executor
foreman   2266  4.3  0.5 2224420 361448 ?      Sl   10:55   0:45 dynflow_executor
foreman   2266  4.2  0.5 2224420 362072 ?      Sl   10:55   0:46 dynflow_executor
foreman   2266  4.2  0.5 2224420 363532 ?      Sl   10:55   0:47 dynflow_executor
foreman   2266  4.1  0.5 2224420 363684 ?      Sl   10:55   0:48 dynflow_executor
foreman   2266  4.1  0.5 2224420 364068 ?      Sl   10:55   0:49 dynflow_executor
foreman   2266  4.0  0.5 2224420 364720 ?      Sl   10:55   0:50 dynflow_executor
foreman   2266  4.0  0.5 2224420 367256 ?      Sl   10:55   0:51 dynflow_executor
foreman   2266  4.0  0.5 2224420 369168 ?      Sl   10:55   0:52 dynflow_executor
foreman   2266  4.1  0.6 2226592 418600 ?      Sl   10:55   0:54 dynflow_executor
foreman   2266  5.0  0.7 2292128 475572 ?      Sl   10:55   1:06 dynflow_executor
foreman   2266  5.6  0.7 2289956 475992 ?      Sl   10:55   1:16 dynflow_executor
foreman   2266  6.1  0.7 2289956 476084 ?      Sl   10:55   1:24 dynflow_executor
foreman   2266  6.8  0.7 2289956 482516 ?      Sl   10:55   1:34 dynflow_executor
foreman   2266  7.4  0.7 2289956 482896 ?      Sl   10:55   1:42 dynflow_executor
foreman   2266  8.5  0.8 2358020 541964 ?      Sl   10:55   2:00 dynflow_executor
foreman   2266  9.6  0.8 2358020 548760 ?      Sl   10:55   2:17 dynflow_executor
foreman   2266 10.2  0.8 2358020 550188 ?      Sl   10:55   2:27 dynflow_executor
foreman   2266 10.9  0.8 2358020 551960 ?      Sl   10:55   2:39 dynflow_executor
foreman   2266 11.6  0.9 2358020 597992 ?      Sl   10:55   2:50 dynflow_executor
foreman   2266 12.6  0.9 2358020 598136 ?      Sl   10:55   3:07 dynflow_executor
foreman   2266 13.4  0.9 2425388 628604 ?      Sl   10:55   3:21 dynflow_executor
foreman   2266 14.2  0.9 2429988 642172 ?      Sl   10:55   3:35 dynflow_executor
foreman   2266 14.8  1.0 2495524 665116 ?      Sl   10:55   3:45 dynflow_executor
foreman   2266 15.4  1.0 2495524 695004 ?      Sl   10:55   3:57 dynflow_executor
foreman   2266 16.1  1.0 2490468 704028 ?      Sl   10:55   4:09 dynflow_executor
foreman   2266 16.1  1.0 2490468 705884 ?      Sl   10:55   4:11 dynflow_executor
---- Stop Syncs ----
foreman   2266 16.0  1.0 2490468 709196 ?      Sl   10:55   4:12 dynflow_executor
foreman   2266 15.8  1.0 2490468 715508 ?      Sl   10:55   4:13 dynflow_executor
foreman   2266 15.8  1.0 2490468 716196 ?      Sl   10:55   4:14 dynflow_executor
foreman   2266 15.5  1.0 2490468 721588 ?      Sl   10:55   4:15 dynflow_executor
foreman   2266 15.2  1.1 2490468 728080 ?      Sl   10:55   4:16 dynflow_executor
Comment 17 errata-xmlrpc 2016-12-19 08:18:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2958
Note You need to log in before you can comment on or make changes to this bug.