Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 749495 - On 16 CPU box installer sets 16 thin processes
Summary: On 16 CPU box installer sets 16 thin processes
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Packaging
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: Unspecified
Assignee: Lukas Zapletal
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: katello-blockers
TreeView+ depends on / blocked
 
Reported: 2011-10-27 08:58 UTC by Lukas Zapletal
Modified: 2019-09-26 13:22 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-22 18:03:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 745652 0 unspecified CLOSED Installation will configure itself to use 1 thin process if only one processor 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 757176 0 unspecified CLOSED Thin process count is set to 0 2021-02-22 00:41:40 UTC

Internal Links: 745652 757176

Description Lukas Zapletal 2011-10-27 08:58:06 UTC 
I did some testing today on a 16 cpu-core box and it turns out that 17 thin instances can fill 2 gigs very quickly with a little help of pulp and candlepin processes.

After system tests are done every thin process consumes about 250 MB of memory from which about 30-100 MB of memory is not swapped (in use) if I read the top utility correctly. And please note our cli system tests are not performance tests or something - its like 150 individual API calls. No loops or something like that.

The box has 2 gigs of memory and it is swapping hardly. That sounds to me like we should have limit a maximum amount of thin processes to either a constant number or better

floor ( ((total_mem - reserve) / (cpu_count * 250m)) )

where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4).

ps - there is a nice bug - if you edit thin.yml and lower the number to let's say N there and then issue "service katello restart" it only restarts first N processes leaving the others still running :-) working workflow should be like:

# service katello stop
# vim /etc/katello/thin.yml
# service katello start

11022 tomcat    20   0 7426m 296m 3684 S  0.0 14.9   0:52.73 java
10059 katello   20   0  249m 108m 2448 S  0.0  5.5   0:16.02 thin
 9802 katello   20   0  245m 106m 2536 S  0.0  5.3   0:15.43 thin
 9742 katello   20   0  245m 105m 2440 S  2.0  5.3   0:16.18 thin
 9887 katello   20   0  245m 103m 2468 S  0.0  5.2   0:14.23 thin
10023 katello   20   0  245m 103m 2364 S  0.3  5.2   0:14.05 thin
 9738 katello   20   0  245m 101m 2532 S  0.3  5.1   0:15.54 thin
10103 katello   20   0  245m  97m 2532 S  0.0  4.9   0:13.66 thin
10091 katello   20   0  245m  96m 2436 S  0.0  4.9   0:14.56 thin
 9763 katello   20   0  245m  94m 2460 S  0.0  4.7   0:14.65 thin
 9771 katello   20   0  245m  84m 2428 S  0.0  4.2   0:17.32 thin
 9995 katello   20   0  248m  51m 2444 S  0.0  2.6   0:16.08 thin
10063 katello   20   0  245m  48m 2440 S  0.0  2.4   0:14.62 thin
 9789 katello   20   0  245m  43m 2520 S  0.0  2.2   0:15.62 thin
 9767 katello   20   0  244m  39m 2424 S  0.3  2.0   0:15.36 thin
 9751 katello   20   0  244m  31m 2364 S  0.0  1.6   0:16.19 thin
Comment 1 Lukas Zapletal 2011-11-02 10:16:53 UTC 
d40a607 749495 - installation script process count
Comment 2 Lukas Zapletal 2011-11-04 09:30:14 UTC 
Pushing one more fix, calculation is now based on number of cpus + 1 and only if it exceeds the recommended number of processes based on total memory, its set to this limit.

In the puppet log there are various information shown:

notice: Scope(Class[Katello::Params]): CPU count: 4
notice: Scope(Class[Katello::Params]): Thin processes recommendation: 5
notice: Scope(Class[Katello::Params]): Total memory: 2104533975.04
notice: Scope(Class[Katello::Params]): Thin consumes: 230000000
notice: Scope(Class[Katello::Params]): Reserve: 500000000
notice: Scope(Class[Katello::Params]): Maximum processes: 6
notice: Scope(Class[Katello::Params]): Thin processes: 5

In this cenario Thin processes is set to five and server has enough memory to handle even more (6) processes.

On a 16 core machine with less memory Thin processess should not be set to 17, but lower number.

2e8e28b 749495 - fix for the total memory calculation
Comment 3 Lukas Zapletal 2011-11-30 09:41:31 UTC 
Please CLOSE this one

https://bugzilla.redhat.com/show_bug.cgi?id=745652#c12
Comment 4 Garik Khachikyan 2011-11-30 09:51:28 UTC 
so seems the approach is:

<snip>
floor ( ((total_mem - reserve) / (cpu_count * 250m)) )
where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4).
</snip>
Comment 6 Garik Khachikyan 2011-11-30 10:06:38 UTC 
version of packages:
katello-0.1.117-1.el6.x86_64
Comment 7 Lukas Zapletal 2011-11-30 10:15:02 UTC 
I have modified the original equotation (from the mail).

Its defined here:

puppet/lib/puppet/parser/functions/katello_process_count.rb

    max_processes = (((total_mem - reserve) / consumes)).floor

So again, the process count is:

CPU COUNT + 1 (IF ENOUGH MEMORY)

Can be less. See the output of puppet main.log:

^[[0;36mnotice: Scope(Class[Katello::Params]): CPU count: 4^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes recommendation: 5^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Total memory: 4015794421.76^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin consumes: 230000000^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Reserve: 500000000^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Maximum processes: 15^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes: 5^[[0m

In your case its 4 CPU box, 5 processes have been set. Maximum recommended processes that this box memory could hold is 15. I don't think it make sense to set more thin processes than CPU + 1 count. Ruby is slow.
Comment 8 Garik Khachikyan 2011-11-30 10:59:09 UTC 
then ON_QA 

thanks Lukas for details there.
Comment 9 Garik Khachikyan 2011-11-30 10:59:28 UTC 
and - verified.
Comment 11 Corey Welton 2012-02-21 17:34:23 UTC 
Also note docs bug #795873 that arose from this.
Note You need to log in before you can comment on or make changes to this bug.