When the environment is under high load, there is a race condition where a httpd graceful restart will be called while the vhost configs are being modified when using the openshift-origin-frontend-apache-vhost which causes the httpd.worker thread to end in a bad state.

We followed up with the httpd team and they traced it down to effectively this:


[Information provided by the httpd dev team]
1. OpenShift edits httpd configuration and calls "httpd.worker -k graceful" to gracefully restart httpd to load new configuration. But for some reason, the confiuration supplied to httpd by OpenShift at the time "httpd -k graceful" is executed is broken. You can see this from the httpd error_log:

> [Tue Sep 23 23:48:02 2014] [notice] SIGUSR1 received.  Doing graceful restart
> httpd.worker: Syntax error on line 222 of /etc/httpd/conf/httpd.conf: Syntax
> error on line 75 of /etc/httpd/conf.d/000000_default.conf: Could not open
> configuration file /etc/httpd/conf.d/openshift/54223d5203ef640de1000981_nagiosmonitor_0_chkexsrv1.conf: No such file or directory

Note that this is also the first date/time when /var/log/messages msg appears:

> Sep 23 23:48:04 ex-std-node3 root: httpd -k graceful already running, perhaps force restart httpd

2. After the Syntax Error during graceful restart, httpd stops itself, so no "httpd.worker" process exists on the system. This is expected behaviour when Syntax Error happens because of invalid configuration files.

3. Next execution of "httpd.worker -k graceful" finds outs that httpd process does not exist and starts the new one. This is expected behaviour. This httpd process handles requests normally and you can see this process in ps output as "httpd.worker -k graceful".