Bug 1976728 - Getting http 500 internal server error due to "ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds" [NEEDINFO]
Summary: Getting http 500 internal server error due to "ActiveRecord::ConnectionTimeou...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installer
Version: 6.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-28 05:49 UTC by Hao Chang Yu
Modified: 2023-08-12 00:29 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
jbhatia: needinfo? (ehelms)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 33974 0 Normal New Getting http 500 internal server error due to "ActiveRecord::ConnectionTimeoutError: could not obtain a connection from ... 2022-07-11 18:26:15 UTC
Red Hat Issue Tracker SAT-16381 0 None None None 2023-03-09 00:40:14 UTC
Red Hat Knowledge Base (Solution) 6964062 0 None None None 2022-07-26 19:16:31 UTC

Description Hao Chang Yu 2021-06-28 05:49:19 UTC
Description of problem:
Getting the following error on default 2 workers 16 threads Puma configurations when sending more than 16 query requests at the same time.
----------------------------------------------
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use
----------------------------------------------

The error is gone after settings the connection pools to 32 (number of worker * number of threads)

/usr/share/foreman/config/database.yml
# Database is managed by foreman::database::postgresql
production:
  adapter: postgresql
  database: foreman
  username: foreman
  password: *********
  pool: 32  <==============

Steps to Reproduce:
1. On any remote host or Satellite itself, run the command to send many concurrent requests to the web server using irb console

irb
require 'rest_client'
50.times { Thread.new { begin; RestClient::Resource.new("https://satellite.example.com/api/v2/hosts?installed_package_name=kernel&page=1&per_page=200", user: "admin", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end } }

2. On Satellite, tail the /var/log/foreman/production.log


Actual results:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use

Requests getting 500 internal server error

Expected results:
No error

Additional info:
I thought each worker process has a separate connection pool but somehow it seems like it is not. Not sure.

Comment 18 wclark 2021-11-23 03:34:44 UTC
Created redmine issue https://projects.theforeman.org/issues/33974 from this bug

Comment 26 Jayant Bhatia 2023-08-04 01:09:45 UTC
Hello Eric,

Do we have any update on this BZ if we are targeting this for Satellite-6.14 GA?


Note You need to log in before you can comment on or make changes to this bug.