Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 749495

Summary: On 16 CPU box installer sets 16 thin processes
Product: Red Hat Satellite Reporter: Lukas Zapletal <lzap>
Component: PackagingAssignee: Lukas Zapletal <lzap>
Status: CLOSED CURRENTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.0.0CC: cwelton, gkhachik
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-22 18:03:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 747354    

Description Lukas Zapletal 2011-10-27 08:58:06 UTC
I did some testing today on a 16 cpu-core box and it turns out that 17 thin instances can fill 2 gigs very quickly with a little help of pulp and candlepin processes.

After system tests are done every thin process consumes about 250 MB of memory from which about 30-100 MB of memory is not swapped (in use) if I read the top utility correctly. And please note our cli system tests are not performance tests or something - its like 150 individual API calls. No loops or something like that.

The box has 2 gigs of memory and it is swapping hardly. That sounds to me like we should have limit a maximum amount of thin processes to either a constant number or better

floor ( ((total_mem - reserve) / (cpu_count * 250m)) )

where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4).

ps - there is a nice bug - if you edit thin.yml and lower the number to let's say N there and then issue "service katello restart" it only restarts first N processes leaving the others still running :-) working workflow should be like:

# service katello stop
# vim /etc/katello/thin.yml
# service katello start

11022 tomcat    20   0 7426m 296m 3684 S  0.0 14.9   0:52.73 java
10059 katello   20   0  249m 108m 2448 S  0.0  5.5   0:16.02 thin
 9802 katello   20   0  245m 106m 2536 S  0.0  5.3   0:15.43 thin
 9742 katello   20   0  245m 105m 2440 S  2.0  5.3   0:16.18 thin
 9887 katello   20   0  245m 103m 2468 S  0.0  5.2   0:14.23 thin
10023 katello   20   0  245m 103m 2364 S  0.3  5.2   0:14.05 thin
 9738 katello   20   0  245m 101m 2532 S  0.3  5.1   0:15.54 thin
10103 katello   20   0  245m  97m 2532 S  0.0  4.9   0:13.66 thin
10091 katello   20   0  245m  96m 2436 S  0.0  4.9   0:14.56 thin
 9763 katello   20   0  245m  94m 2460 S  0.0  4.7   0:14.65 thin
 9771 katello   20   0  245m  84m 2428 S  0.0  4.2   0:17.32 thin
 9995 katello   20   0  248m  51m 2444 S  0.0  2.6   0:16.08 thin
10063 katello   20   0  245m  48m 2440 S  0.0  2.4   0:14.62 thin
 9789 katello   20   0  245m  43m 2520 S  0.0  2.2   0:15.62 thin
 9767 katello   20   0  244m  39m 2424 S  0.3  2.0   0:15.36 thin
 9751 katello   20   0  244m  31m 2364 S  0.0  1.6   0:16.19 thin

Comment 1 Lukas Zapletal 2011-11-02 10:16:53 UTC
d40a607 749495 - installation script process count

Comment 2 Lukas Zapletal 2011-11-04 09:30:14 UTC
Pushing one more fix, calculation is now based on number of cpus + 1 and only if it exceeds the recommended number of processes based on total memory, its set to this limit.

In the puppet log there are various information shown:

notice: Scope(Class[Katello::Params]): CPU count: 4
notice: Scope(Class[Katello::Params]): Thin processes recommendation: 5
notice: Scope(Class[Katello::Params]): Total memory: 2104533975.04
notice: Scope(Class[Katello::Params]): Thin consumes: 230000000
notice: Scope(Class[Katello::Params]): Reserve: 500000000
notice: Scope(Class[Katello::Params]): Maximum processes: 6
notice: Scope(Class[Katello::Params]): Thin processes: 5

In this cenario Thin processes is set to five and server has enough memory to handle even more (6) processes.

On a 16 core machine with less memory Thin processess should not be set to 17, but lower number.

2e8e28b 749495 - fix for the total memory calculation

Comment 3 Lukas Zapletal 2011-11-30 09:41:31 UTC
Please CLOSE this one

https://bugzilla.redhat.com/show_bug.cgi?id=745652#c12

Comment 4 Garik Khachikyan 2011-11-30 09:51:28 UTC
so seems the approach is:

<snip>
floor ( ((total_mem - reserve) / (cpu_count * 250m)) )
where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4).
</snip>

Comment 6 Garik Khachikyan 2011-11-30 10:06:38 UTC
version of packages:
katello-0.1.117-1.el6.x86_64

Comment 7 Lukas Zapletal 2011-11-30 10:15:02 UTC
I have modified the original equotation (from the mail).

Its defined here:

puppet/lib/puppet/parser/functions/katello_process_count.rb

    max_processes = (((total_mem - reserve) / consumes)).floor

So again, the process count is:

CPU COUNT + 1 (IF ENOUGH MEMORY)

Can be less. See the output of puppet main.log:

^[[0;36mnotice: Scope(Class[Katello::Params]): CPU count: 4^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes recommendation: 5^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Total memory: 4015794421.76^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin consumes: 230000000^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Reserve: 500000000^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Maximum processes: 15^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes: 5^[[0m

In your case its 4 CPU box, 5 processes have been set. Maximum recommended processes that this box memory could hold is 15. I don't think it make sense to set more thin processes than CPU + 1 count. Ruby is slow.

Comment 8 Garik Khachikyan 2011-11-30 10:59:09 UTC
then ON_QA 

thanks Lukas for details there.

Comment 9 Garik Khachikyan 2011-11-30 10:59:28 UTC
and - verified.

Comment 11 Corey Welton 2012-02-21 17:34:23 UTC
Also note docs bug #795873 that arose from this.