Created attachment 1154127 [details] foreman debug Description of problem: Have around 12000 content hosts registered already to satellite/capsule ( each capsule has 1k hosts). This is not only noticed during 12k. It was noticed very early stages also. Started registering in batches of 50 concurrently. This is noticed occasionally. Noticed below error from UI: --- This website is under heavy load We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later. --- Passenger-status seem to have 100 requests in queue for long. Its not reducing at all. passenger-status Version : 4.0.18 Date : 2016-05-05 04:18:25 -0400 Instance: 9784 ----------- General information ----------- Max pool size : 6 Processes : 6 Requests in top-level queue : 0 ----------- Application groups ----------- /usr/share/foreman#default: App root: /usr/share/foreman Requests in queue: 100 * PID: 10663 Sessions: 1 Processed: 127 Uptime: 2h 0m 40s CPU: 0% Memory : 656M Last used: 1h 9m 5s * PID: 15583 Sessions: 1 Processed: 13 Uptime: 1h 8m 55s CPU: 0% Memory : 257M Last used: 1h 8m 50s * PID: 15612 Sessions: 1 Processed: 26 Uptime: 1h 8m 55s CPU: 0% Memory : 512M Last used: 1h 7m 0s * PID: 15641 Sessions: 1 Processed: 11 Uptime: 1h 8m 54s CPU: 0% Memory : 264M Last used: 1h 8m 46s * PID: 16329 Sessions: 1 Processed: 22 Uptime: 1h 7m 0s CPU: 0% Memory : 484M Last used: 1h 6m 41s /etc/puppet/rack#default: App root: /etc/puppet/rack Requests in queue: 0 * PID: 11430 Sessions: 0 Processed: 20 Uptime: 1h 52m 24s CPU: 0% Memory : 50M Last used: 22m 23s Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. register content hosts to capsule concurrently in batches ( 50 per batch) with some interval between each batch 2. 3. Actual results: UI fails to open Expected results: UI should open Additional info:
Are you able to see which http call during registration is taking a long time? I looked at the foreman debug but had some trouble figuring this out. It may be useful to enable the apache scoreboard as well, which will give the same info.
(In reply to Chris Duryee from comment #3) > Are you able to see which http call during registration is taking a long > time? I looked at the foreman debug but had some trouble figuring this out. > > It may be useful to enable the apache scoreboard as well, which will give > the same info. Hello Chris, do you have experience with enabling that? I have added: ExtendedStatus On <Location "/server-status"> SetHandler server-status ###Require host example.com </Location> and restarted httpd (restarted whole Sat) but when accessing http(s)://<fqdn>/server-status, I'm getting passenger's default 404 page (well, not sure it is passenger's, but it is definitely not httpd's default page).
Here is how I did it: in new file '/etc/httpd/conf.modules.d/status.load': LoadModule status_module modules/mod_status.so I suspect your existing config will work after creating this status.load file. On my sat, I also created /etc/httpd/conf.d/server-status.conf: ExtendedStatus On <Location "/server-status"> SetHandler server-status </Location> and restarted httpd.
closing as duplicate, other bug is an RFE to allow setting # of passenger workers via installer option which I think will fix this. *** This bug has been marked as a duplicate of bug 1163452 ***