Bug 1337525 - httpd will not restart after reseting vmdb
Summary: httpd will not restart after reseting vmdb
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.5.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: GA
: 5.6.0
Assignee: Nick Carboni
QA Contact: luke couzens
URL:
Whiteboard: appliance:database
: 1342292 (view as bug list)
Depends On:
Blocks: 1310121
TreeView+ depends on / blocked
 
Reported: 2016-05-19 11:45 UTC by Alex Krzos
Modified: 2017-08-30 08:04 UTC (History)
8 users (show)

Fixed In Version: 5.6.0.12
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-29 16:04:17 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1348 0 normal SHIPPED_LIVE CFME 5.6.0 bug fixes and enhancement update 2016-06-29 18:50:04 UTC

Description Alex Krzos 2016-05-19 11:45:06 UTC
Description of problem:
Occassionaly httpd will refuse to start after reseting the vmdb.  This is causing a lot of my performance tests which require reseting the database to a clean state to run correctly.



Version-Release number of selected component (if applicable):
5.5 appliances

How reproducible:
Intermittent, sometimes resets will go smooth 10-20 of them and then once httpd refuses to start it requires an entire reboot to allow it to start again.

Steps to Reproduce:
systemctl stop evmserverd
systemctl restart rh-postgresql94-postgresql
vmdb
bin/rake evm:dbsync:local_uninstall
DISABLE_DATABASE_ENVIRONMENT_CHECK=1 bin/rake evm:db:reset
bin/rake db:seed
systemctl start evmserverd

* evmserverd starts, httpd just refuses to start

Actual results:
# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-05-19 07:31:59 EDT; 4min 16s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 21293 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 59893 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
  Process: 21291 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 21291 (code=exited, status=1/FAILURE)

May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Starting The Apache HTTP Server...
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 httpd[21291]: [Thu May 19 07:31:59.907315 2016] [so:warn] [pid 21291] AH01574: module ssl_module is already loaded, skipping
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 httpd[21291]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 kill[21293]: kill: cannot find process ""
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service: control process exited, code=exited status=1
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Failed to start The Apache HTTP Server.
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Unit httpd.service entered failed state.
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service failed.
Expected results:
httpd to start

Additional info:

miq_apache.log contents:

[Thu May 19 07:36:26.728960 2016] [core:notice] [pid 21484] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu May 19 07:36:26.729562 2016] [suexec:notice] [pid 21484] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 19 07:36:26.741616 2016] [so:warn] [pid 21484] AH01574: module ssl_module is already loaded, skipping
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
[Thu May 19 07:36:26.743301 2016] [auth_digest:notice] [pid 21484] AH01757: generating secret for digest authentication ...
[Thu May 19 07:36:26.743797 2016] [lbmethod_heartbeat:notice] [pid 21484] AH02282: No slotmem from mod_heartmonitor
[Thu May 19 07:36:26.743856 2016] [proxy_balancer:emerg] [pid 21484] (22)Invalid argument: AH01185: worker slotmem_create failed
[Thu May 19 07:36:26.743863 2016] [:emerg] [pid 21484] AH00020: Configuration Failed, exiting

Comment 2 Alex Krzos 2016-05-19 12:08:02 UTC
I found a work around, If I clear /run/httpd/ I can finally restart httpd without restarting the appliance.

Comment 3 Joe Rafaniello 2016-06-15 20:15:50 UTC
*** Bug 1342292 has been marked as a duplicate of this bug. ***

Comment 4 Nick Carboni 2016-06-15 20:46:52 UTC
https://github.com/ManageIQ/manageiq/pull/9236

Comment 5 CFME Bot 2016-06-16 16:06:05 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/01634e35036ec29032ae1fae2f6976ea11d70052

commit 01634e35036ec29032ae1fae2f6976ea11d70052
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Jun 15 16:37:55 2016 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Jun 15 16:37:55 2016 -0400

    Gracefully stop httpd when starting the server
    
    Previously we would `killall -9 httpd` when starting the server.
    This would leave around some IPC information in `/run/httpd`
    which could prevent the httpd service from coming up cleanly again
    with the error message "AH01179: balancer slotmem_create failed"
    in the miq_apache.log file.
    
    This changes `MiqServer.prep_apache_proxying` to use a graceful
    shutdown instead.
    This method does not leave any files around in `/run/httpd`.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1337525

 app/models/miq_server/environment_management.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 6 Oleg Barenboim 2016-06-16 16:51:20 UTC
PR merged and backported to Darga

Comment 8 amogh 2016-06-28 22:18:31 UTC
verified for 30 iterations to cleanup_appliance and check for http service status, 
issue not reproduced on 5.6.0.13.20160624114606_13a9153.

Comment 10 errata-xmlrpc 2016-06-29 16:04:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348


Note You need to log in before you can comment on or make changes to this bug.