Bug 1405533

Summary: puppet requests blocking UI
Product: Red Hat Satellite Reporter: Bryan Kearney <bkearney>
Component: InstallerAssignee: Stephen Benjamin <stbenjam>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.0.4CC: akrzos, bbuckingham, bkearney, cdonnell, cduryee, cwelton, dgupte, egolov, ehelms, jcallaha, mburgerh, mmccune, mmello, mrichter, mtenheuv, oshtaier, perfbz, psuriset, rakumar, stbenjam, xdmoon
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1163452 Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1163452, 1472734    
Bug Blocks: 1115190    
Attachments:
Description Flags
slow load time
none
verification screenshot none

Comment 3 jcallaha 2017-10-25 17:57:36 UTC
This doesn't seem to be working properly in Satellite 6.3 Beta Snap 21. I used the same steps I did from https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c26 and loading even the sync status page took 2.7 minutes (see attached).

Although there are 12 active passengers, it looks like only one is processing requests.

-bash-4.2# passenger-status 
*** Cleaning stale folder /var/run/rubygem-passenger/passenger.1.0.19520
Version : 4.0.18
Date    : 2017-10-25 13:46:22 -0400
Instance: 22662
----------- General information -----------
Max pool size : 12
Processes     : 12
Requests in top-level queue : 0

----------- Application groups -----------
/usr/share/foreman#default:
  App root: /usr/share/foreman
  Requests in queue: 0
  * PID: 27516   Sessions: 0       Processed: 1900    Uptime: 10h 49m 38s
    CPU: 1%      Memory  : 784M    Last used: 5s ago

/etc/puppet/rack#default:
  App root: /etc/puppet/rack
  Requests in queue: 0
  * PID: 23275   Sessions: 0       Processed: 476     Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 39M     Last used: 3m 34s 
  * PID: 23282   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 14M     Last used: 47h 11m
  * PID: 23289   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 8M      Last used: 47h 11m
  * PID: 23296   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 8M      Last used: 47h 11m
  * PID: 23303   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 8M      Last used: 47h 11m
  * PID: 23310   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 4M      Last used: 47h 11m
  * PID: 23317   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 5M      Last used: 47h 11m
  * PID: 23324   Sessions: 0       Processed: 0       Uptime: 47h 11m 46s
    CPU: 0%      Memory  : 5M      Last used: 47h 11m
  * PID: 23331   Sessions: 0       Processed: 0       Uptime: 47h 11m 45s
    CPU: 0%      Memory  : 5M      Last used: 47h 11m
  * PID: 23338   Sessions: 0       Processed: 0       Uptime: 47h 11m 45s
    CPU: 0%      Memory  : 5M      Last used: 47h 11m
  * PID: 29249   Sessions: 0       Processed: 0       Uptime: 47h 4m 11s
    CPU: 0%      Memory  : 5M      Last used: 47h 4m 1



-bash-4.2# cat /etc/httpd/conf.modules.d/passenger_extra.conf
# The Passenger Apache module configuration file is being
# managed by Puppet and changes will be overwritten.
<IfModule mod_passenger.c>
  PassengerMaxPoolSize 12
  PassengerMaxInstancesPerApp 6
  PassengerMaxRequestQueueSize 250
  PassengerMaxRequests 10000
  PassengerStatThrottleRate 120
</IfModule>

Comment 4 jcallaha 2017-10-25 17:59:11 UTC
Created attachment 1343360 [details]
slow load time

Comment 5 Satellite Program 2017-10-25 18:22:27 UTC
Upstream bug assigned to stbenjam

Comment 6 Eric Helms 2017-10-26 14:32:13 UTC
I believe this requires https://bugzilla.redhat.com/show_bug.cgi?id=1472734 in order to prevent Puppet passenger from choking out the passenger pool. I will attempt to test to double check.

Comment 8 jcallaha 2017-11-01 21:00:37 UTC
Verified in Satellite 6.3 snap 22.

Continued with the same testing process as above. This time, everything is working as expected. The number of passengers started small, but grew to match the demand. While kept under 12 concurrent requests, the page loaded quickly. See attached screenshot.

Before
------
-bash-4.2# passenger-status 
Version : 4.0.18
Date    : 2017-11-01 16:53:52 -0400
Instance: 32372
----------- General information -----------
Max pool size : 12
Processes     : 3
Requests in top-level queue : 0

----------- Application groups -----------
/usr/share/foreman#default:
  App root: /usr/share/foreman
  Requests in queue: 0
  * PID: 15920   Sessions: 0       Processed: 41      Uptime: 1h 39m 38s
    CPU: 0%      Memory  : 549M    Last used: 6s ago
  * PID: 24898   Sessions: 0       Processed: 0       Uptime: 1m 37s
    CPU: 0%      Memory  : 115M    Last used: 1m 37s ago

/etc/puppet/rack#default:
  App root: /etc/puppet/rack
  Requests in queue: 0
  * PID: 479     Sessions: 0       Processed: 31      Uptime: 3h 9m 14s
    CPU: 0%      Memory  : 49M     Last used: 11m 37s a





During
------
-bash-4.2# passenger-status 
Version : 4.0.18
Date    : 2017-11-01 16:55:58 -0400
Instance: 32372
----------- General information -----------
Max pool size : 12
Processes     : 12
Requests in top-level queue : 0

----------- Application groups -----------
/usr/share/foreman#default:
  App root: /usr/share/foreman
  Requests in queue: 0
  * PID: 15920   Sessions: 1       Processed: 44      Uptime: 1h 41m 44s
    CPU: 0%      Memory  : 570M    Last used: 15s ago
  * PID: 24898   Sessions: 1       Processed: 2       Uptime: 3m 43s
    CPU: 22%     Memory  : 575M    Last used: 14s ago
  * PID: 25093   Sessions: 1       Processed: 2       Uptime: 1m 53s
    CPU: 45%     Memory  : 593M    Last used: 15s ago
  * PID: 25119   Sessions: 1       Processed: 8       Uptime: 1m 53s
    CPU: 47%     Memory  : 668M    Last used: 15s ago
  * PID: 25147   Sessions: 1       Processed: 2       Uptime: 1m 52s
    CPU: 43%     Memory  : 638M    Last used: 13s ago
  * PID: 25172   Sessions: 1       Processed: 2       Uptime: 1m 52s
    CPU: 45%     Memory  : 602M    Last used: 14s ago
  * PID: 25199   Sessions: 1       Processed: 16      Uptime: 1m 51s
    CPU: 10%     Memory  : 274M    Last used: 13s ago
  * PID: 25237   Sessions: 1       Processed: 2       Uptime: 1m 44s
    CPU: 11%     Memory  : 271M    Last used: 15s ago
  * PID: 25276   Sessions: 1       Processed: 1       Uptime: 1m 40s
    CPU: 10%     Memory  : 257M    Last used: 13s ago
  * PID: 25423   Sessions: 1       Processed: 1       Uptime: 8s
    CPU: 87%     Memory  : 230M    Last used: 8s ago
  * PID: 25451   Sessions: 0       Processed: 1       Uptime: 6s
    CPU: 29%     Memory  : 125M    Last used: 6s ago

/etc/puppet/rack#default:
  App root: /etc/puppet/rack
  Requests in queue: 0
  * PID: 479     Sessions: 0       Processed: 31      Uptime: 3h 11m 20s
    CPU: 0%      Memory  : 49M     Last used: 13m 43s

Comment 9 jcallaha 2017-11-01 21:01:04 UTC
Created attachment 1346749 [details]
verification screenshot

Comment 10 Bryan Kearney 2018-02-21 17:32:53 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336

Comment 11 Bryan Kearney 2018-02-21 17:33:18 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336