Description of problem: Occassionaly httpd will refuse to start after reseting the vmdb. This is causing a lot of my performance tests which require reseting the database to a clean state to run correctly. Version-Release number of selected component (if applicable): 5.5 appliances How reproducible: Intermittent, sometimes resets will go smooth 10-20 of them and then once httpd refuses to start it requires an entire reboot to allow it to start again. Steps to Reproduce: systemctl stop evmserverd systemctl restart rh-postgresql94-postgresql vmdb bin/rake evm:dbsync:local_uninstall DISABLE_DATABASE_ENVIRONMENT_CHECK=1 bin/rake evm:db:reset bin/rake db:seed systemctl start evmserverd * evmserverd starts, httpd just refuses to start Actual results: # systemctl start httpd Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. # systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2016-05-19 07:31:59 EDT; 4min 16s ago Docs: man:httpd(8) man:apachectl(8) Process: 21293 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE) Process: 59893 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS) Process: 21291 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) Main PID: 21291 (code=exited, status=1/FAILURE) May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Starting The Apache HTTP Server... May 19 07:31:59 CF-R0000-DB-Benchmark-5541 httpd[21291]: [Thu May 19 07:31:59.907315 2016] [so:warn] [pid 21291] AH01574: module ssl_module is already loaded, skipping May 19 07:31:59 CF-R0000-DB-Benchmark-5541 httpd[21291]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE May 19 07:31:59 CF-R0000-DB-Benchmark-5541 kill[21293]: kill: cannot find process "" May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service: control process exited, code=exited status=1 May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Failed to start The Apache HTTP Server. May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Unit httpd.service entered failed state. May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service failed. Expected results: httpd to start Additional info: miq_apache.log contents: [Thu May 19 07:36:26.728960 2016] [core:notice] [pid 21484] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Thu May 19 07:36:26.729562 2016] [suexec:notice] [pid 21484] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu May 19 07:36:26.741616 2016] [so:warn] [pid 21484] AH01574: module ssl_module is already loaded, skipping AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message [Thu May 19 07:36:26.743301 2016] [auth_digest:notice] [pid 21484] AH01757: generating secret for digest authentication ... [Thu May 19 07:36:26.743797 2016] [lbmethod_heartbeat:notice] [pid 21484] AH02282: No slotmem from mod_heartmonitor [Thu May 19 07:36:26.743856 2016] [proxy_balancer:emerg] [pid 21484] (22)Invalid argument: AH01185: worker slotmem_create failed [Thu May 19 07:36:26.743863 2016] [:emerg] [pid 21484] AH00020: Configuration Failed, exiting
I found a work around, If I clear /run/httpd/ I can finally restart httpd without restarting the appliance.
*** Bug 1342292 has been marked as a duplicate of this bug. ***
https://github.com/ManageIQ/manageiq/pull/9236
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/01634e35036ec29032ae1fae2f6976ea11d70052 commit 01634e35036ec29032ae1fae2f6976ea11d70052 Author: Nick Carboni <ncarboni> AuthorDate: Wed Jun 15 16:37:55 2016 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Wed Jun 15 16:37:55 2016 -0400 Gracefully stop httpd when starting the server Previously we would `killall -9 httpd` when starting the server. This would leave around some IPC information in `/run/httpd` which could prevent the httpd service from coming up cleanly again with the error message "AH01179: balancer slotmem_create failed" in the miq_apache.log file. This changes `MiqServer.prep_apache_proxying` to use a graceful shutdown instead. This method does not leave any files around in `/run/httpd`. https://bugzilla.redhat.com/show_bug.cgi?id=1337525 app/models/miq_server/environment_management.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
PR merged and backported to Darga
https://github.com/ManageIQ/manageiq/pull/7620
verified for 30 iterations to cleanup_appliance and check for http service status, issue not reproduced on 5.6.0.13.20160624114606_13a9153.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1348