Bug 844123

Until we can start using mod_express (with a more modern version of apache) I suspect we'll need to detect and auto correct this issue.  an httpd reload might do it, an httpd restart might as well.  If that doesn't work we'll have to manually kill the graceful's and issue a reload.

Another thought is to have stickshift issue a restart when it detects a graceful is already running.

At the least, we should add a lock to the restart_httpd_graceful function that cartridges use; maby through an external script.  That should take care of the dominant cause of the issue being observed.

Created attachment 601344 [details]
Daemon server to coalesce 'graceful' restarts of apache

Created attachment 601345 [details]
Service command to be run from cartridges for graceful restart

Proposed solution :
   Have a daemon that serves the 'graceful' requests, that queues up and coalesces the requests. If there is a 'graceful' process running, and another 'N' applications want a graceful restart, this method will ensure that one 'graceful' addresses all 'N' at once. 
Locks may be inefficient as they will still run N+1 requests in the above scenario (versus 2).

We don't want an entire daemon that sits and watches this.

(In reply to comment #2)
> At the least, we should add a lock to the restart_httpd_graceful function
> that cartridges use; maby through an external script.  That should take care
> of the dominant cause of the issue being observed.

So this seems to be that a graceful (which I think is non blocking) just sits in the process queue until intervention is had.

Would the lock mean that no applications could be created until after the graceful/lock clears?  I'm not sure if the multiple graceful's are a symptom or the problem.

(In reply to comment #6)
> We don't want an entire daemon that sits and watches this.

Why not? 
Its more efficient than locks, and it reduces the number of 'graceful' processes that end up running also.
It is not watching on any process, just queuing up the requests on a single socket bind.

Sorry - noticed the additional comments on this ticket after I wrote and started testing the lock script.

Queuing up the graceful requests and servicing them all with one call is a lot more efficient; as we discovered when the idler was re-written to do that.

The benefit of processing them once-per-application with a wrapper is its really simple.

Neither of these solutions or what we do now gives you clarity which application may have broken the Apache configuration when there's a lot of simultaneous reconfiguration.

I'm going to finish testing the lock script and issue a pull request for it.  If it leads to restart times which are too long; then we should look at making the wrapper script coalesce requests into one configuration check and graceful restart.

open pull request

Pull requests:
https://github.com/openshift/li/pull/152
https://github.com/openshift/crankcase/pull/302

Pull requests accepted.

Verified this bug on devenv_1931, and PASS.

On old instance (devenv_stage_226), this issue can be reproduced.
1. Create 10 apps at the same time.
2. In instance, run the following command to watch http restart process.
<--snip-->
------
root     26113 25202  0 08:38 ?        00:00:00 /bin/sh /sbin/service httpd graceful
root     26121 26113  0 08:38 ?        00:00:00 /bin/bash /etc/init.d/httpd graceful
root     26129 26121  0 08:38 ?        00:00:00 /bin/sh /usr/sbin/apachectl graceful
root     26148 25245  0 08:38 ?        00:00:00 /bin/sh /sbin/service httpd graceful
root     26160 26148  0 08:38 ?        00:00:00 /bin/bash /etc/init.d/httpd graceful
root     26166 26129  4 08:38 ?        00:00:00 /usr/sbin/httpd -k graceful
root     26167 26160  0 08:38 ?        00:00:00 /bin/sh /usr/sbin/apachectl graceful
------
<--snip-->

Found that at least 2 "/etc/init.d/httpd gracesful" processe are running in the same time.


On new instance (devenv_1931), do the same operation like the above steps:
# while :; do ps -ef|grep grace|grep -v grep;sleep 1; echo ------; done
<--snip-->
------
root      1568   712  6 08:58 ?        00:00:00 ruby /usr/libexec/stickshift/cartridges/abstract/info/bin/httpd_singular graceful
root      1790  1568  0 08:58 ?        00:00:00 /bin/sh /sbin/service httpd graceful
root      1801  1790  0 08:58 ?        00:00:00 /bin/bash /etc/init.d/httpd graceful
root      1804  1801  0 08:58 ?        00:00:00 /bin/sh /usr/sbin/apachectl graceful
------
root      1568   712  6 08:58 ?        00:00:00 ruby /usr/libexec/stickshift/cartridges/abstract/info/bin/httpd_singular graceful
root      1790  1568  0 08:58 ?        00:00:00 /bin/sh /sbin/service httpd graceful
root      1801  1790  0 08:58 ?        00:00:00 /bin/bash /etc/init.d/httpd graceful
root      1804  1801  0 08:58 ?        00:00:00 /bin/sh /usr/sbin/apachectl graceful
------
root      1568   712  6 08:58 ?        00:00:00 ruby /usr/libexec/stickshift/cartridges/abstract/info/bin/httpd_singular graceful
root      1790  1568  0 08:58 ?        00:00:00 /bin/sh /sbin/service httpd graceful
root      1801  1790  0 08:58 ?        00:00:00 /bin/bash /etc/init.d/httpd graceful
root      1804  1801  0 08:58 ?        00:00:00 /bin/sh /usr/sbin/apachectl graceful
root      1832  1804  0 08:58 ?        00:00:00 /usr/sbin/httpd -k graceful
------
root      1568   712  6 08:58 ?        00:00:00 ruby /usr/libexec/stickshift/cartridges/abstract/info/bin/httpd_singular graceful
root      1790  1568  0 08:58 ?        00:00:00 /bin/sh /sbin/service httpd graceful
root      1801  1790  0 08:58 ?        00:00:00 /bin/bash /etc/init.d/httpd graceful
root      1804  1801  0 08:58 ?        00:00:00 /bin/sh /usr/sbin/apachectl graceful
root      1832  1804  0 08:58 ?        00:00:00 /usr/sbin/httpd -k graceful
------
<--snip-->

Found that no multiple "/etc/init.d/httpd gracesful" processes are running in the same time.