Bug 830286

Summary:	rhc-auto-idler is spiking the load on the ex-nodes during run...
Product:	OKD	Reporter:	Thomas Wiest <twiest>
Component:	Containers	Assignee:	Rob Millner <rmillner>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	mfisher, qgong, xtian
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-06-25 18:27:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Thomas Wiest 2012-06-08 18:20:41 UTC

Description of problem:
An unintended consequence of fixing bug 824546 is that now rhc-autoidler runs very fast (doesn't have to wait for the curl's to finish), which causes httpd to be restarted many times in a row, which spikes the load of the machine.

A possible fix for this would be to have rhc-idler have a --no-httpd-restart flag, and have rhc-autoidler call into rhc-idler with that flag and then once rhc-autoidler had finished idling all apps, it restarted httpd itself.



Version-Release number of selected component (if applicable):
rhc-node-0.93.18-1.el6_2.x86_64


How reproducible:
very in PROD


Steps to Reproduce:
1. use rhc-autoidler to idle a bunch of apps
2. notice that it restarts httpd after each idle
3. notice that the load on the system goes up because of all of the restarts
 

Actual results:
restarts apache after each idle


Expected results:
would be best for rhc-autoidler to just restart apache once after all of the rhc-idler calls.

Comment 1 Rob Millner 2012-06-13 02:32:54 UTC

Commit 89c0705 moves the httpd restart until the end.

This creates a small race condition where the application is shutting down and httpd doesn't know it should be restarted.  Believed to be an acceptable trade off since the app has to be idle for a long time anyway for this logic to kick in and the client can always retry.

Comment 2 Thomas Wiest 2012-06-20 20:26:49 UTC

This bug is not fixed.

When trying to run rhc-idler using the -n option I get the following:

/usr/bin/rhc-idler: illegal option -- n
Usage: /usr/bin/rhc-idler
  -u UUID       idles the app
  -l            list all idled apps
  -n            don't restart httpd


I tracked this down to this line (notice it's missing an 'n').

while getopts 'u:dl' OPTION

Comment 3 Rob Millner 2012-06-20 21:27:28 UTC

Li commit 47579a7.

# rhc-idler -u 54ef45a02dc1483b874169d9cd045c1e -n
Stopping 54ef45a02dc1483b874169d9cd045c1e...
    jenkins_ctl.sh                                         [  OK  ]

Comment 4 Xiaoli Tian 2012-06-21 10:34:00 UTC

Verified it on devenv_1861
rhc-idler has -n option to not restart httpd

[root@ip-10-99-27-79 ~]# rhc-idler -h
/usr/bin/rhc-idler: illegal option -- h
Usage: /usr/bin/rhc-idler
  -u UUID       idles the app
  -l            list all idled apps
  -n            don't restart httpd


[root@ip-10-99-27-79 ~]# rhc-idler -u 8aa2984bc6844e8f80c5796b3ce1ca50 -n 
Stopping 8aa2984bc6844e8f80c5796b3ce1ca50...
    phpjenkins_ctl.sh                                      [  OK  ]

Thomas, please see if it works for you now, thanks

Comment 5 Thomas Wiest 2012-06-21 13:47:48 UTC

This seems to be working now in INT.