Bug 815490

Summary:	"systemctl status httpd.service" says 'failed' after issuing a graceful Apache restart
Product:	[Fedora] Fedora	Reporter:	Joe Honton <joe>
Component:	httpd	Assignee:	Joe Orton <jorton>
Status:	CLOSED WORKSFORME	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	16	CC:	jkaluza, jorton, pahan
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-04-23 21:20:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joe Honton 2012-04-23 17:40:11 UTC

Description of problem:

Beginning with Fedora 15, the command "service httpd status" is handled by the
new systemd commands instead of the older SysVinit commands. 

The new systemd commands work well to start, stop and display the status of
httpd, but it does not support the graceful option.

As a workaround, I can still use the old command to gracefully reload the
configuration files, and this works: the server reloads the configuration and
continues to serve requests, but unfortunately the status command says
otherwise.

Version-Release number of selected component (if applicable):

Apache httpd 2.2.22

How reproducible:

Start the Apache service using the systemd commands; then restart using the SysVinit commands which allow the 'graceful' option; then check the status using the new systemd commands.

Steps to Reproduce:
$ systemctl start httpd.service
$ service httpd graceful
$ systemctl status httpd.service

Actual results:

$ systemctl status httpd.service
 httpd.service - The Apache HTTP Server (prefork MPM)
 Loaded: loaded (/lib/systemd/system/httpd.service; enabled)
 Active: failed since Fri, 20 Apr 2012 14:05:39 -0700; 4h 44min ago
 Process: 20019 ExecStop=/usr/sbin/httpd $OPTIONS -k stop (code=exited,status=0/SUCCESS)
 Process: 26544 ExecReload=/bin/kill -HUP $MAINPID (code=exited,status=0/SUCCESS)
 Process: 26543 ExecReload=/usr/sbin/httpd $OPTIONS -t (code=exited,status=0/SUCCESS)
 Main PID: 1114 (code=exited, status=1/FAILURE)
 CGroup: name=systemd:/system/httpd.service

Expected results:

When properly working the third line should read as:
 Active: active (running since Fri, 20 Apr 2012 14:05:39 -0700; 4h 44min ago

Additional info:

Jeff Trawick at Apache says, "The report appears to be limited to scripts provided with Fedora, so report this to the Fedora project." 

Apache bugzilla https://issues.apache.org/bugzilla/show_bug.cgi?id=53117

Comment 1 Joe Orton 2012-04-23 18:22:03 UTC

I can't reproduce that with httpd-2.2.22-2.fc16.x86_64

# rpm -q httpd
# ls -li /etc/httpd/run/httpd.pid /var/run/httpd/httpd.pid
# grep -i pidfile /etc/httpd/conf/httpd.conf

Comment 2 Joe Orton 2012-04-23 18:22:28 UTC

I meant to say... please run those commands and post the output.

Comment 3 Joe Honton 2012-04-23 18:50:27 UTC

# rpm -q httpd
httpd-2.2.22-1.fc16.x86_64

# ls -li /etc/httpd/run/httpd.pid /var/run/httpd/httpd.pid
13970 -rw-r--r--. 1 root root 6 Apr 22 03:13 /etc/httpd/run/httpd.pid
13970 -rw-r--r--. 1 root root 6 Apr 22 03:13 /var/run/httpd/httpd.pid

# grep -i pidfile /etc/httpd/conf/httpd.conf
PidFile run/httpd.pid

Comment 4 Joe Orton 2012-04-23 19:12:55 UTC

Does the httpd parent process pid (look at "ps axf") match up with the pidfile in that file?  Is /etc/httpd/run a symlink to /var/run/httpd?

The most likely reason I can think of to explain why systemd thinks the service is dead is if it is really running is some kind of mismatch between the pidfiles.

Comment 5 Joe Honton 2012-04-23 20:57:34 UTC

Your troubleshooting has led me to investigate the httpd error log which revealed an unclean shutdown leading to the PID file being overwritten. Here's the clues:

[Fri Apr 20 14:05:39 2012] [notice] Graceful restart requested, doing restart
[Fri Apr 20 14:05:39 2012] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Fri Apr 20 14:05:56 2012] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Fri Apr 20 14:05:56 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Apr 20 14:05:56 2012] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Apr 20 14:05:56 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.0g-fips configured -- resuming normal operations

When I run the graceful command now, everything works as expected. I am satisfied with this explanation.

Comment 6 Joe Orton 2012-04-23 21:20:08 UTC

OK, great.  Thanks for the update.