Bug 906321

Summary:	systemd stop/restart httpd needs ~90 secs
Product:	[Fedora] Fedora	Reporter:	Valerio Pulese <valerio>
Component:	httpd	Assignee:	Joe Orton <jorton>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	18	CC:	erikb, jkaluza, jorton, jtripcevic, leho, pahan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	httpd-2.4.4-3.fc18	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-04-25 13:55:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Valerio Pulese 2013-01-31 13:08:27 UTC

Description of problem:
systemd stop/restart httpd needs ~90 secs
There are no pending requests (web server is quiescent).

Version-Release number of selected component (if applicable):
2.4

How reproducible:
systemctl stop httpd
(or systemctl restart httpd)

Steps to Reproduce:
1. systemctl stop httpd &
2. 'systemctl status httpd' or 'pgrep httpd' show some httpd orphans
3.
  
Actual results:
'systemctl stop httpd' hang for ~90 secs (default TimeoutStopSec=90 secs  reached)

Expected results:
Apache should stop after few secs even on a graceful stop/restart.

Additional info:
http://httpd.apache.org/docs/2.4/stopping.html
"...you shouldn't ever need to send signals to any process except the parent."
but the default KillMode for systemd is "control-group".

workaround: Add KillMode=none to /usr/lib/systemd/system/httpd.service
An alternative workaround could be to set  TimeoutStopSec=10 or so
but KillMode=none seems to me the polite way.

Comment 1 Joe Orton 2013-02-13 16:22:09 UTC

Hmmmm, interesting.  This isn't trivially reproducible here.  Does the error_log say anything about why the daemon is slow to shut down?

"systemctl stop httpd" *will* first send attempt to terminate (only) the httpd parent, not the whole process group.  The question is the correct behaviour in the case of a parent which is slow to terminate.  We are using graceful-stop here which complicates things.

Comment 2 Valerio Pulese 2013-02-14 11:38:20 UTC

With KillMode=none
Feb 14 11:02:28 webmail1 systemd[1]: Stopping The Apache HTTP Server...
Feb 14 11:02:28 webmail1 systemd[1]: Stopped The Apache HTTP Server.
With default KillMode=control-group
Feb 14 11:07:11 webmail1 systemd[1]: Stopping The Apache HTTP Server...
Feb 14 11:08:42 webmail1 systemd[1]: httpd.service stopping timed out (2). Killing.
Feb 14 11:08:42 webmail1 systemd[1]: Stopped The Apache HTTP Server.
Feb 14 11:08:42 webmail1 systemd[1]: Unit httpd.service entered failed state

/var/log/http/error_log
With KillMode=none
[Thu Feb 14 11:02:28.717716 2013] [mpm_prefork:notice] [pid 10224] AH00170: caught SIGWINCH, shutting down gracefully

With default KillMode=control-group there's a stack dump

Thu Feb 14 11:07:11.924371 2013] [mpm_prefork:notice] [pid 10267] AH00170: caught SIGWINCH, shutting down gracefully
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f99df359e90 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f99df359e90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x359707ca8e)[0x7f99dc160a8e]
/lib64/libapr-1.so.0(apr_pool_destroy+0x19c)[0x7f99dc8d585c]
/etc/httpd/modules/mod_mpm_prefork.so(+0x31be)[0x7f99d37d81be]
/etc/httpd/modules/mod_mpm_prefork.so(+0x39fb)[0x7f99d37d89fb]
======= Backtrace: =========
/lib64/libpthread.so.0(+0x3597c0f000)[0x7f99dc6af000]
/etc/httpd/modules/mod_authn_file.so(+0xbc0)[0x7f99da64cbc0]
======= Memory map: ========
[...]

I did a 'yum upgrade' and kernel was updated from 3.7.4-204.fc18.x86_64 to 3.7.6-201.fc18.x86_64 (systemd and httpd are already the latest)
Now I don't get the stack dump anymore.

Other  details:
# systemctl start httpd.service
# ps -def|grep http
root      1752  1151  0 12:18 pts/0    00:00:00 tail -f /var/log/httpd/error_log
root      1847     1  4 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1849  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1850  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1851  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1852  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1853  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root      1855  1151  0 12:22 pts/0    00:00:00 grep --color=auto http
# systemctl status httpd.service 
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: active (running) since Thu 2013-02-14 12:22:43 CET; 14s ago
         Process: 1833 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
        Main PID: 1847 (httpd)
          Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
          CGroup: name=systemd:/system/httpd.service
                  ├─1847 /usr/sbin/httpd -DFOREGROUND
                  ├─1849 /usr/sbin/httpd -DFOREGROUND
                  ├─1850 /usr/sbin/httpd -DFOREGROUND
                  ├─1851 /usr/sbin/httpd -DFOREGROUND
                  ├─1852 /usr/sbin/httpd -DFOREGROUND
                  └─1853 /usr/sbin/httpd -DFOREGROUND

Feb 14 12:22:41 webmail1 systemd[1]: Starting The Apache HTTP Server...
Feb 14 12:22:43 webmail1 systemd[1]: Started The Apache HTTP Server.
# systemctl stop httpd.service 
..zzz...

The error_log instantaneously:
[Thu Feb 14 12:23:09.204078 2013] [mpm_prefork:notice] [pid 1847] AH00170: caught SIGWINCH, shutting down gracefully

In the meantime:
root      1752  1151  0 12:18 pts/0    00:00:00 tail -f /var/log/httpd/error_log
apache    1851     1  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1852     1  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root      1857  1151  0 12:23 pts/0    00:00:00 systemctl stop httpd.service
root      1858  1857  0 12:23 pts/0    00:00:00 /usr/bin/systemd-tty-ask-password-agent --watch

Before you ask : yes there is SSL virtual host but the certificate is not password encrypted. Anyway why systemd-tty-ask-password-agent should ask for a password stopping httpd :-) ?

Comment 3 jean-filip tripcevic 2013-03-31 13:56:08 UTC

For what it's worth I have the same problem on latest F18 (Kernel 3.8.4).
I put the workaround in (KillMode=none) and problem goes away.


[Sun Mar 31 15:34:43.215636 2013] [mpm_prefork:notice] [pid 676] AH00170: caught SIGWINCH, shutting down gracefully
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb9306548 ***
======= Backtrace: =========
/lib/libc.so.6(+0x47b17ff9)[0xb7295ff9]
/lib/libapr-1.so.0(apr_pool_destroy+0x1ca)[0xb740d10a]
/etc/httpd/modules/mod_mpm_prefork.so(+0x27cf)[0xb70367cf]
/etc/httpd/modules/mod_mpm_prefork.so(+0x318b)[0xb703718b]
[0xb7705400]
/etc/httpd/modules/mod_authn_dbd.so(+0x9f9)[0xb71ac9f9]
/lib/ld-linux.so.2(+0x47a89d04)[0xb7715d04]
/lib/libc.so.6(+0x47ad3f01)[0xb7251f01]

Comment 4 Fedora Update System 2013-04-16 12:43:36 UTC

httpd-2.4.4-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/httpd-2.4.4-4.fc19

Comment 5 Fedora Update System 2013-04-16 16:12:11 UTC

Package httpd-2.4.4-4.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing httpd-2.4.4-4.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-5797/httpd-2.4.4-4.fc19
then log in and leave karma (feedback).

Comment 6 Fedora Update System 2013-04-25 13:55:12 UTC

httpd-2.4.4-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Fedora Update System 2013-05-17 12:41:35 UTC

httpd-2.4.4-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/httpd-2.4.4-3.fc18

Comment 8 Fedora Update System 2013-05-28 01:05:54 UTC

httpd-2.4.4-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.