Bug 749495

Summary:	On 16 CPU box installer sets 16 thin processes
Product:	Red Hat Satellite	Reporter:	Lukas Zapletal <lzap>
Component:	Packaging	Assignee:	Lukas Zapletal <lzap>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Katello QA List <katello-qa-list>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	6.0.0	CC:	cwelton, gkhachik
Target Milestone:	Unspecified
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-08-22 18:03:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	747354

Description Lukas Zapletal 2011-10-27 08:58:06 UTC

I did some testing today on a 16 cpu-core box and it turns out that 17 thin instances can fill 2 gigs very quickly with a little help of pulp and candlepin processes.

After system tests are done every thin process consumes about 250 MB of memory from which about 30-100 MB of memory is not swapped (in use) if I read the top utility correctly. And please note our cli system tests are not performance tests or something - its like 150 individual API calls. No loops or something like that.

The box has 2 gigs of memory and it is swapping hardly. That sounds to me like we should have limit a maximum amount of thin processes to either a constant number or better

floor ( ((total_mem - reserve) / (cpu_count * 250m)) )

where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4).

ps - there is a nice bug - if you edit thin.yml and lower the number to let's say N there and then issue "service katello restart" it only restarts first N processes leaving the others still running :-) working workflow should be like:

# service katello stop
# vim /etc/katello/thin.yml
# service katello start

11022 tomcat    20   0 7426m 296m 3684 S  0.0 14.9   0:52.73 java
10059 katello   20   0  249m 108m 2448 S  0.0  5.5   0:16.02 thin
 9802 katello   20   0  245m 106m 2536 S  0.0  5.3   0:15.43 thin
 9742 katello   20   0  245m 105m 2440 S  2.0  5.3   0:16.18 thin
 9887 katello   20   0  245m 103m 2468 S  0.0  5.2   0:14.23 thin
10023 katello   20   0  245m 103m 2364 S  0.3  5.2   0:14.05 thin
 9738 katello   20   0  245m 101m 2532 S  0.3  5.1   0:15.54 thin
10103 katello   20   0  245m  97m 2532 S  0.0  4.9   0:13.66 thin
10091 katello   20   0  245m  96m 2436 S  0.0  4.9   0:14.56 thin
 9763 katello   20   0  245m  94m 2460 S  0.0  4.7   0:14.65 thin
 9771 katello   20   0  245m  84m 2428 S  0.0  4.2   0:17.32 thin
 9995 katello   20   0  248m  51m 2444 S  0.0  2.6   0:16.08 thin
10063 katello   20   0  245m  48m 2440 S  0.0  2.4   0:14.62 thin
 9789 katello   20   0  245m  43m 2520 S  0.0  2.2   0:15.62 thin
 9767 katello   20   0  244m  39m 2424 S  0.3  2.0   0:15.36 thin
 9751 katello   20   0  244m  31m 2364 S  0.0  1.6   0:16.19 thin

Comment 1 Lukas Zapletal 2011-11-02 10:16:53 UTC

d40a607 749495 - installation script process count

Comment 2 Lukas Zapletal 2011-11-04 09:30:14 UTC

Pushing one more fix, calculation is now based on number of cpus + 1 and only if it exceeds the recommended number of processes based on total memory, its set to this limit.

In the puppet log there are various information shown:

notice: Scope(Class[Katello::Params]): CPU count: 4
notice: Scope(Class[Katello::Params]): Thin processes recommendation: 5
notice: Scope(Class[Katello::Params]): Total memory: 2104533975.04
notice: Scope(Class[Katello::Params]): Thin consumes: 230000000
notice: Scope(Class[Katello::Params]): Reserve: 500000000
notice: Scope(Class[Katello::Params]): Maximum processes: 6
notice: Scope(Class[Katello::Params]): Thin processes: 5

In this cenario Thin processes is set to five and server has enough memory to handle even more (6) processes.

On a 16 core machine with less memory Thin processess should not be set to 17, but lower number.

2e8e28b 749495 - fix for the total memory calculation

Comment 3 Lukas Zapletal 2011-11-30 09:41:31 UTC

Please CLOSE this one

https://bugzilla.redhat.com/show_bug.cgi?id=745652#c12

Comment 4 Garik Khachikyan 2011-11-30 09:51:28 UTC

so seems the approach is:

<snip>
floor ( ((total_mem - reserve) / (cpu_count * 250m)) )
where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4).
</snip>

Comment 6 Garik Khachikyan 2011-11-30 10:06:38 UTC

version of packages:
katello-0.1.117-1.el6.x86_64

Comment 7 Lukas Zapletal 2011-11-30 10:15:02 UTC

I have modified the original equotation (from the mail).

Its defined here:

puppet/lib/puppet/parser/functions/katello_process_count.rb

    max_processes = (((total_mem - reserve) / consumes)).floor

So again, the process count is:

CPU COUNT + 1 (IF ENOUGH MEMORY)

Can be less. See the output of puppet main.log:

^[[0;36mnotice: Scope(Class[Katello::Params]): CPU count: 4^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes recommendation: 5^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Total memory: 4015794421.76^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin consumes: 230000000^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Reserve: 500000000^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Maximum processes: 15^[[0m
^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes: 5^[[0m

In your case its 4 CPU box, 5 processes have been set. Maximum recommended processes that this box memory could hold is 15. I don't think it make sense to set more thin processes than CPU + 1 count. Ruby is slow.

Comment 8 Garik Khachikyan 2011-11-30 10:59:09 UTC

then ON_QA 

thanks Lukas for details there.

Comment 9 Garik Khachikyan 2011-11-30 10:59:28 UTC

and - verified.

Comment 11 Corey Welton 2012-02-21 17:34:23 UTC

Also note docs bug #795873 that arose from this.