Bug 830286 - rhc-auto-idler is spiking the load on the ex-nodes during run...
rhc-auto-idler is spiking the load on the ex-nodes during run...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Rob Millner
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-08 14:20 EDT by Thomas Wiest
Modified: 2015-05-14 18:55 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-25 14:27:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas Wiest 2012-06-08 14:20:41 EDT
Description of problem:
An unintended consequence of fixing bug 824546 is that now rhc-autoidler runs very fast (doesn't have to wait for the curl's to finish), which causes httpd to be restarted many times in a row, which spikes the load of the machine.

A possible fix for this would be to have rhc-idler have a --no-httpd-restart flag, and have rhc-autoidler call into rhc-idler with that flag and then once rhc-autoidler had finished idling all apps, it restarted httpd itself.



Version-Release number of selected component (if applicable):
rhc-node-0.93.18-1.el6_2.x86_64


How reproducible:
very in PROD


Steps to Reproduce:
1. use rhc-autoidler to idle a bunch of apps
2. notice that it restarts httpd after each idle
3. notice that the load on the system goes up because of all of the restarts
 

Actual results:
restarts apache after each idle


Expected results:
would be best for rhc-autoidler to just restart apache once after all of the rhc-idler calls.
Comment 1 Rob Millner 2012-06-12 22:32:54 EDT
Commit 89c0705 moves the httpd restart until the end.

This creates a small race condition where the application is shutting down and httpd doesn't know it should be restarted.  Believed to be an acceptable trade off since the app has to be idle for a long time anyway for this logic to kick in and the client can always retry.
Comment 2 Thomas Wiest 2012-06-20 16:26:49 EDT
This bug is not fixed.

When trying to run rhc-idler using the -n option I get the following:

/usr/bin/rhc-idler: illegal option -- n
Usage: /usr/bin/rhc-idler
  -u UUID       idles the app
  -l            list all idled apps
  -n            don't restart httpd


I tracked this down to this line (notice it's missing an 'n').

while getopts 'u:dl' OPTION
Comment 3 Rob Millner 2012-06-20 17:27:28 EDT
Li commit 47579a7.

# rhc-idler -u 54ef45a02dc1483b874169d9cd045c1e -n
Stopping 54ef45a02dc1483b874169d9cd045c1e...
    jenkins_ctl.sh                                         [  OK  ]
Comment 4 Xiaoli Tian 2012-06-21 06:34:00 EDT
Verified it on devenv_1861
rhc-idler has -n option to not restart httpd

[root@ip-10-99-27-79 ~]# rhc-idler -h
/usr/bin/rhc-idler: illegal option -- h
Usage: /usr/bin/rhc-idler
  -u UUID       idles the app
  -l            list all idled apps
  -n            don't restart httpd


[root@ip-10-99-27-79 ~]# rhc-idler -u 8aa2984bc6844e8f80c5796b3ce1ca50 -n 
Stopping 8aa2984bc6844e8f80c5796b3ce1ca50...
    phpjenkins_ctl.sh                                      [  OK  ]

Thomas, please see if it works for you now, thanks
Comment 5 Thomas Wiest 2012-06-21 09:47:48 EDT
This seems to be working now in INT.

Note You need to log in before you can comment on or make changes to this bug.