Bug 2025811

Description of problem:

After the auto-tuning of Puma Threads and Workers were introduced in Satellite 6.9.6 and above via https://bugzilla.redhat.com/show_bug.cgi?id=1998309 (Parent BZ# 1998291) , Several Satellite users have complained about servers hitting OOM during Daily Repo sync or CV publish+promote , Just because pulp or postgres or mongo can not get enough memory to process the tasks and around half of the system memory is consumed by Puma workers.

Version-Release number of selected component (if applicable):

Satellite 6.9.6 and above

Satellite 6.10


How reproducible:

Always [For Satellite customers]


Steps to Reproduce:
1. Install a Satellite 6.8 , having 8 vCPU and 32 GB ram
2. Enable and Sync 30 big + small repos. Create few CVs using the same.
3. Create a daily sync plan for all those repos, to run around midnight.

   A) Note down overall Memory usage during each of those activities. 

4. Upgrade the same satellite to Satellite 6.9.6 or 6.9.7 , and observe how the satellite is behaving during the same activities. Specifically monitor the memory usage by foreman.service.

5. Upgrade the same satellite to Satellite 6.10 and repeat the same monitoring. Check how much memory is being occupied by foreman.service and postgresql service.
 

Actual results:

Puma worker count increased to 1.5 times of the vCPU count and that consumes lot of system resources causing other Pulp related services to not able to get enough memory to complete their tasks at certain times and results in OOM.


Expected results:

The puma workers value needs to be tuned in a way that It should not overconsume the system memory and should not be the cause of OOM. Both CPU and Memory should be part of same calculation or else, let's just do this:

If vCPU count is upto 12, then set the count of workers as the value of $nproc.

If the vCPU count is 16 or above, always default the value to 12 or 14.

Obviously, It does not has to be the same values if we have a better formula with us.


Additional info:

Since 6.9.6, Installer automatically sets the count of Puma threads to 5 and Puma workers to the 1.5 * $nproc

For workers, It basically calculates two values based on CPU and Memory and which ever is the lowest, It choses that as the value to use. 

      min(
        $facts['processors']['count'] * 1.5,
        ($facts['memory']['system']['total_bytes']/(1024 * 1024 * 1024)) - 1.5
      )

If I am not wrong, 100% of the cases, the second formula with memory will not matter as that will always be higher and the installer will by default set the worker value as the (1.5 * $nproc).

For a 16 vCPU satellite : 24 workers

For an 8 vCPU satellite : 12 workers.


Each worker can consume from 750 MB to 1.5 GB memory on average (never seen going beyond 2 GB) and if you calculate in that way then,

For a 16 vCPU satellite : 24 workers can consume around 18 GB to 35 GB system memory.

For an 8 vCPU satellite : 12 workers can consume around 9 to 18 GB system memory.


So if just these components will consume that much system memory, then Just try to think what will happen with other services or processes here like Postgres, mongodb, qdrouterd, puppetserver, tomcat, etc which are known to require some good amount of memory to function properly as well.

It's not that every single customer will allocate 50+ GB memory for a 16 vCPU satellite. In many scenarios, You will see 8 vCPU or 16 vCPU systems having 32 GB or 20 GB or 40 GB or 74 GB memory allocated.  

Again for a Customer having 64 GB memory with 16 vCPUs, It puma workers are consuming 35 GB memory just of themselves, that is 54% of total system memory consumed already.