Bug 1976728

Description of problem:
Getting the following error on default 2 workers 16 threads Puma configurations when sending more than 16 query requests at the same time.
----------------------------------------------
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use
----------------------------------------------

The error is gone after settings the connection pools to 32 (number of worker * number of threads)

/usr/share/foreman/config/database.yml
# Database is managed by foreman::database::postgresql
production:
  adapter: postgresql
  database: foreman
  username: foreman
  password: *********
  pool: 32  <==============

Steps to Reproduce:
1. On any remote host or Satellite itself, run the command to send many concurrent requests to the web server using irb console

irb
require 'rest_client'
50.times { Thread.new { begin; RestClient::Resource.new("https://satellite.example.com/api/v2/hosts?installed_package_name=kernel&page=1&per_page=200", user: "admin", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end } }

2. On Satellite, tail the /var/log/foreman/production.log


Actual results:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use

Requests getting 500 internal server error

Expected results:
No error

Additional info:
I thought each worker process has a separate connection pool but somehow it seems like it is not. Not sure.