Bug 1976694

Description of problem:
Firstly, I can't find the best component that either matching the web server or Puma. Feel free to move it to a suitable component if you found one.

Puma won't queue requests or not queue fast enough even the backlog is low if we send some requests that are enough to make all the workers busy (no more available threads) for awhile. Some requests will then begin to timeout after about 75 seconds (6 tcp syn retries I think) and getting connection reset by Puma. After that the client will get either 502 Bad Gateway or 503 service not available.

Can refer to the below links for more information:
----------------------------------------
https://github.com/puma/puma/issues/1405
https://github.com/puma/puma/pull/1278 <=== Before this PR, each worker would accept and queue as many connections without limit. If I rollback this PR, all 502 and 503 are error are gone.
----------------------------------------

Another reason of 503 service unavailable is caused by insufficient database connections in the active record when sending more than 16 query requests at the same time. I will open another bugzilla for this.
----------------------------------------
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use
----------------------------------------

Steps to Reproduce:
1. On any remote host or Satellite itself, run the command to send many concurrent requests to the web server using irb console

irb
require 'rest_client'
200.times { Thread.new { begin; RestClient::Resource.new("https://satellite.example.com/api/v2/hosts?installed_package_name=kernel&page=1&per_page=500", user: "admin", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end } }


2. On another window, run puma-status (need to install this gem manually) to check the status.

while true; do scl enable tfm 'puma-status /usr/share/foreman/tmp/puma.state'; sleep 3; done

17938 (/usr/share/foreman/tmp/puma.state) Uptime: 17m28s | Phase: 0 | Load: 32[████████████████████████████████]32
 └ 27171 CPU: 100.0% Mem: 1126 MB Uptime:  3m25s | Load: 16[████████████████]16 Queue: 1  <==== Will see either low or no queue
 └ 27177 CPU:  93.3% Mem: 1228 MB Uptime:  3m25s | Load: 16[████████████████]16    <==== Will see either low or no queue


3. On another window, tail the error log and you should see  the following error after around 75 seconds.

tail -f /var/log/httpd/foreman-ssl_error_ssl.log

[proxy_http:error] [pid 3466] (104)Connection reset by peer: [client xx.xx.xx.xx:xxxxx] AH01102: error reading status line from remote server 127.0.0.1:3000
[proxy:error] [pid 3466] [client xx.xx.xx.xx:xxxxx] AH00898: Error reading from remote server returned by /api/v2/hosts
[proxy_http:error] [pid 3484] (104)Connection reset by peer: [client xx.xx.xx.xx:xxxxx] AH01102: error reading status line from remote server 127.0.0.1:3000
[proxy:error] [pid 3484] [client xx.xx.xx.xx:xxxxx] AH00898: Error reading from remote server returned by /api/v2/hosts
[proxy_http:error] [pid 3492] (104)Connection reset by peer: [client xx.xx.xx.xx:xxxxx] AH01102: error reading status line from remote server 127.0.0.1:3000
[proxy:error] [pid 3492] [client xx.xx.xx.xx:xxxxx] AH00898: Error reading from remote server returned by /api/v2/hosts
[proxy:error] [pid 3494] (110)Connection timed out: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[proxy:error] [pid 3494] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
[proxy_http:error] [pid 3494] [client xx.xx.xx.xx:xxxxx] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
[proxy:error] [pid 3483] (110)Connection timed out: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[proxy:error] [pid 3483] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
[proxy_http:error] [pid 3483] [client xx.xx.xx.xx:xxxxx] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
[proxy:error] [pid 3481] (110)Connection timed out: AH00957: HTTP: attempt to connect to 127.0.0.1:3000 (127.0.0.1) failed
[proxy:error] [pid 3481] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s

Actual results:
Puma stops accepting connections (queuing requests) even if the queue/backlog is low causing connection timeout and connection reset. It is because the requests are waiting in the socket backlog and timed out.

Expected results:
Worker should queue the requests until certain limit even if all threads are busy.


Additional infos:
This never happened in Passenger.

This might not be considered as a bug (maybe some people will disagree that is is a bug) based on how Puma is designed and it could probably means that the web server just need more workers in order to serve more concurrent long requests and given that each worker has its own backlog queue. Maybe we need to document this in the tuning guide or something?