Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1337525 - httpd will not restart after reseting vmdb
httpd will not restart after reseting vmdb
Status: CLOSED ERRATA
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance (Show other bugs)
5.5.0
Unspecified Unspecified
medium Severity medium
: GA
: 5.6.0
Assigned To: Nick Carboni
luke couzens
appliance:database
:
: 1342292 (view as bug list)
Depends On:
Blocks: 1310121
  Show dependency treegraph
 
Reported: 2016-05-19 07:45 EDT by Alex Krzos
Modified: 2017-08-30 04:04 EDT (History)
8 users (show)

See Also:
Fixed In Version: 5.6.0.12
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-29 12:04:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1348 normal SHIPPED_LIVE CFME 5.6.0 bug fixes and enhancement update 2016-06-29 14:50:04 EDT

  None (edit)
Description Alex Krzos 2016-05-19 07:45:06 EDT
Description of problem:
Occassionaly httpd will refuse to start after reseting the vmdb.  This is causing a lot of my performance tests which require reseting the database to a clean state to run correctly.



Version-Release number of selected component (if applicable):
5.5 appliances

How reproducible:
Intermittent, sometimes resets will go smooth 10-20 of them and then once httpd refuses to start it requires an entire reboot to allow it to start again.

Steps to Reproduce:
systemctl stop evmserverd
systemctl restart rh-postgresql94-postgresql
vmdb
bin/rake evm:dbsync:local_uninstall
DISABLE_DATABASE_ENVIRONMENT_CHECK=1 bin/rake evm:db:reset
bin/rake db:seed
systemctl start evmserverd

* evmserverd starts, httpd just refuses to start

Actual results:
# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-05-19 07:31:59 EDT; 4min 16s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 21293 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 59893 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
  Process: 21291 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 21291 (code=exited, status=1/FAILURE)

May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Starting The Apache HTTP Server...
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 httpd[21291]: [Thu May 19 07:31:59.907315 2016] [so:warn] [pid 21291] AH01574: module ssl_module is already loaded, skipping
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 httpd[21291]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 kill[21293]: kill: cannot find process ""
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service: control process exited, code=exited status=1
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Failed to start The Apache HTTP Server.
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: Unit httpd.service entered failed state.
May 19 07:31:59 CF-R0000-DB-Benchmark-5541 systemd[1]: httpd.service failed.
Expected results:
httpd to start

Additional info:

miq_apache.log contents:

[Thu May 19 07:36:26.728960 2016] [core:notice] [pid 21484] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu May 19 07:36:26.729562 2016] [suexec:notice] [pid 21484] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 19 07:36:26.741616 2016] [so:warn] [pid 21484] AH01574: module ssl_module is already loaded, skipping
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
[Thu May 19 07:36:26.743301 2016] [auth_digest:notice] [pid 21484] AH01757: generating secret for digest authentication ...
[Thu May 19 07:36:26.743797 2016] [lbmethod_heartbeat:notice] [pid 21484] AH02282: No slotmem from mod_heartmonitor
[Thu May 19 07:36:26.743856 2016] [proxy_balancer:emerg] [pid 21484] (22)Invalid argument: AH01185: worker slotmem_create failed
[Thu May 19 07:36:26.743863 2016] [:emerg] [pid 21484] AH00020: Configuration Failed, exiting
Comment 2 Alex Krzos 2016-05-19 08:08:02 EDT
I found a work around, If I clear /run/httpd/ I can finally restart httpd without restarting the appliance.
Comment 3 Joe Rafaniello 2016-06-15 16:15:50 EDT
*** Bug 1342292 has been marked as a duplicate of this bug. ***
Comment 4 Nick Carboni 2016-06-15 16:46:52 EDT
https://github.com/ManageIQ/manageiq/pull/9236
Comment 5 CFME Bot 2016-06-16 12:06:05 EDT
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/01634e35036ec29032ae1fae2f6976ea11d70052

commit 01634e35036ec29032ae1fae2f6976ea11d70052
Author:     Nick Carboni <ncarboni@redhat.com>
AuthorDate: Wed Jun 15 16:37:55 2016 -0400
Commit:     Nick Carboni <ncarboni@redhat.com>
CommitDate: Wed Jun 15 16:37:55 2016 -0400

    Gracefully stop httpd when starting the server
    
    Previously we would `killall -9 httpd` when starting the server.
    This would leave around some IPC information in `/run/httpd`
    which could prevent the httpd service from coming up cleanly again
    with the error message "AH01179: balancer slotmem_create failed"
    in the miq_apache.log file.
    
    This changes `MiqServer.prep_apache_proxying` to use a graceful
    shutdown instead.
    This method does not leave any files around in `/run/httpd`.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1337525

 app/models/miq_server/environment_management.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 6 Oleg Barenboim 2016-06-16 12:51:20 EDT
PR merged and backported to Darga
Comment 8 amogh 2016-06-28 18:18:31 EDT
verified for 30 iterations to cleanup_appliance and check for http service status, 
issue not reproduced on 5.6.0.13.20160624114606_13a9153.
Comment 10 errata-xmlrpc 2016-06-29 12:04:17 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348

Note You need to log in before you can comment on or make changes to this bug.