I did some testing today on a 16 cpu-core box and it turns out that 17 thin instances can fill 2 gigs very quickly with a little help of pulp and candlepin processes. After system tests are done every thin process consumes about 250 MB of memory from which about 30-100 MB of memory is not swapped (in use) if I read the top utility correctly. And please note our cli system tests are not performance tests or something - its like 150 individual API calls. No loops or something like that. The box has 2 gigs of memory and it is swapping hardly. That sounds to me like we should have limit a maximum amount of thin processes to either a constant number or better floor ( ((total_mem - reserve) / (cpu_count * 250m)) ) where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4). ps - there is a nice bug - if you edit thin.yml and lower the number to let's say N there and then issue "service katello restart" it only restarts first N processes leaving the others still running :-) working workflow should be like: # service katello stop # vim /etc/katello/thin.yml # service katello start 11022 tomcat 20 0 7426m 296m 3684 S 0.0 14.9 0:52.73 java 10059 katello 20 0 249m 108m 2448 S 0.0 5.5 0:16.02 thin 9802 katello 20 0 245m 106m 2536 S 0.0 5.3 0:15.43 thin 9742 katello 20 0 245m 105m 2440 S 2.0 5.3 0:16.18 thin 9887 katello 20 0 245m 103m 2468 S 0.0 5.2 0:14.23 thin 10023 katello 20 0 245m 103m 2364 S 0.3 5.2 0:14.05 thin 9738 katello 20 0 245m 101m 2532 S 0.3 5.1 0:15.54 thin 10103 katello 20 0 245m 97m 2532 S 0.0 4.9 0:13.66 thin 10091 katello 20 0 245m 96m 2436 S 0.0 4.9 0:14.56 thin 9763 katello 20 0 245m 94m 2460 S 0.0 4.7 0:14.65 thin 9771 katello 20 0 245m 84m 2428 S 0.0 4.2 0:17.32 thin 9995 katello 20 0 248m 51m 2444 S 0.0 2.6 0:16.08 thin 10063 katello 20 0 245m 48m 2440 S 0.0 2.4 0:14.62 thin 9789 katello 20 0 245m 43m 2520 S 0.0 2.2 0:15.62 thin 9767 katello 20 0 244m 39m 2424 S 0.3 2.0 0:15.36 thin 9751 katello 20 0 244m 31m 2364 S 0.0 1.6 0:16.19 thin
d40a607 749495 - installation script process count
Pushing one more fix, calculation is now based on number of cpus + 1 and only if it exceeds the recommended number of processes based on total memory, its set to this limit. In the puppet log there are various information shown: notice: Scope(Class[Katello::Params]): CPU count: 4 notice: Scope(Class[Katello::Params]): Thin processes recommendation: 5 notice: Scope(Class[Katello::Params]): Total memory: 2104533975.04 notice: Scope(Class[Katello::Params]): Thin consumes: 230000000 notice: Scope(Class[Katello::Params]): Reserve: 500000000 notice: Scope(Class[Katello::Params]): Maximum processes: 6 notice: Scope(Class[Katello::Params]): Thin processes: 5 In this cenario Thin processes is set to five and server has enough memory to handle even more (6) processes. On a 16 core machine with less memory Thin processess should not be set to 17, but lower number. 2e8e28b 749495 - fix for the total memory calculation
Please CLOSE this one https://bugzilla.redhat.com/show_bug.cgi?id=745652#c12
so seems the approach is: <snip> floor ( ((total_mem - reserve) / (cpu_count * 250m)) ) where reserve should be some decent amount of mermory allocated for non-thin processes (something like total_mem / 4). </snip>
version of packages: katello-0.1.117-1.el6.x86_64
I have modified the original equotation (from the mail). Its defined here: puppet/lib/puppet/parser/functions/katello_process_count.rb max_processes = (((total_mem - reserve) / consumes)).floor So again, the process count is: CPU COUNT + 1 (IF ENOUGH MEMORY) Can be less. See the output of puppet main.log: ^[[0;36mnotice: Scope(Class[Katello::Params]): CPU count: 4^[[0m ^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes recommendation: 5^[[0m ^[[0;36mnotice: Scope(Class[Katello::Params]): Total memory: 4015794421.76^[[0m ^[[0;36mnotice: Scope(Class[Katello::Params]): Thin consumes: 230000000^[[0m ^[[0;36mnotice: Scope(Class[Katello::Params]): Reserve: 500000000^[[0m ^[[0;36mnotice: Scope(Class[Katello::Params]): Maximum processes: 15^[[0m ^[[0;36mnotice: Scope(Class[Katello::Params]): Thin processes: 5^[[0m In your case its 4 CPU box, 5 processes have been set. Maximum recommended processes that this box memory could hold is 15. I don't think it make sense to set more thin processes than CPU + 1 count. Ruby is slow.
then ON_QA thanks Lukas for details there.
and - verified.
Also note docs bug #795873 that arose from this.