Bug 906321 - systemd stop/restart httpd needs ~90 secs
Summary: systemd stop/restart httpd needs ~90 secs
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: httpd   
(Show other bugs)
Version: 18
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-01-31 13:08 UTC by Valerio Pulese
Modified: 2015-08-08 23:35 UTC (History)
6 users (show)

Fixed In Version: httpd-2.4.4-3.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-04-25 13:55:10 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description Valerio Pulese 2013-01-31 13:08:27 UTC
Description of problem:
systemd stop/restart httpd needs ~90 secs
There are no pending requests (web server is quiescent).

Version-Release number of selected component (if applicable):
2.4

How reproducible:
systemctl stop httpd
(or systemctl restart httpd)

Steps to Reproduce:
1. systemctl stop httpd &
2. 'systemctl status httpd' or 'pgrep httpd' show some httpd orphans
3.
  
Actual results:
'systemctl stop httpd' hang for ~90 secs (default TimeoutStopSec=90 secs  reached)

Expected results:
Apache should stop after few secs even on a graceful stop/restart.

Additional info:
http://httpd.apache.org/docs/2.4/stopping.html
"...you shouldn't ever need to send signals to any process except the parent."
but the default KillMode for systemd is "control-group".

workaround: Add KillMode=none to /usr/lib/systemd/system/httpd.service
An alternative workaround could be to set  TimeoutStopSec=10 or so
but KillMode=none seems to me the polite way.

Comment 1 Joe Orton 2013-02-13 16:22:09 UTC
Hmmmm, interesting.  This isn't trivially reproducible here.  Does the error_log say anything about why the daemon is slow to shut down?

"systemctl stop httpd" *will* first send attempt to terminate (only) the httpd parent, not the whole process group.  The question is the correct behaviour in the case of a parent which is slow to terminate.  We are using graceful-stop here which complicates things.

Comment 2 Valerio Pulese 2013-02-14 11:38:20 UTC
With KillMode=none
Feb 14 11:02:28 webmail1 systemd[1]: Stopping The Apache HTTP Server...
Feb 14 11:02:28 webmail1 systemd[1]: Stopped The Apache HTTP Server.
With default KillMode=control-group
Feb 14 11:07:11 webmail1 systemd[1]: Stopping The Apache HTTP Server...
Feb 14 11:08:42 webmail1 systemd[1]: httpd.service stopping timed out (2). Killing.
Feb 14 11:08:42 webmail1 systemd[1]: Stopped The Apache HTTP Server.
Feb 14 11:08:42 webmail1 systemd[1]: Unit httpd.service entered failed state

/var/log/http/error_log
With KillMode=none
[Thu Feb 14 11:02:28.717716 2013] [mpm_prefork:notice] [pid 10224] AH00170: caught SIGWINCH, shutting down gracefully

With default KillMode=control-group there's a stack dump

Thu Feb 14 11:07:11.924371 2013] [mpm_prefork:notice] [pid 10267] AH00170: caught SIGWINCH, shutting down gracefully
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f99df359e90 ***
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x00007f99df359e90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x359707ca8e)[0x7f99dc160a8e]
/lib64/libapr-1.so.0(apr_pool_destroy+0x19c)[0x7f99dc8d585c]
/etc/httpd/modules/mod_mpm_prefork.so(+0x31be)[0x7f99d37d81be]
/etc/httpd/modules/mod_mpm_prefork.so(+0x39fb)[0x7f99d37d89fb]
======= Backtrace: =========
/lib64/libpthread.so.0(+0x3597c0f000)[0x7f99dc6af000]
/etc/httpd/modules/mod_authn_file.so(+0xbc0)[0x7f99da64cbc0]
======= Memory map: ========
[...]

I did a 'yum upgrade' and kernel was updated from 3.7.4-204.fc18.x86_64 to 3.7.6-201.fc18.x86_64 (systemd and httpd are already the latest)
Now I don't get the stack dump anymore.

Other  details:
# systemctl start httpd.service
# ps -def|grep http
root      1752  1151  0 12:18 pts/0    00:00:00 tail -f /var/log/httpd/error_log
root      1847     1  4 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1849  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1850  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1851  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1852  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1853  1847  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root      1855  1151  0 12:22 pts/0    00:00:00 grep --color=auto http
# systemctl status httpd.service 
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: active (running) since Thu 2013-02-14 12:22:43 CET; 14s ago
         Process: 1833 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
        Main PID: 1847 (httpd)
          Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
          CGroup: name=systemd:/system/httpd.service
                  ├─1847 /usr/sbin/httpd -DFOREGROUND
                  ├─1849 /usr/sbin/httpd -DFOREGROUND
                  ├─1850 /usr/sbin/httpd -DFOREGROUND
                  ├─1851 /usr/sbin/httpd -DFOREGROUND
                  ├─1852 /usr/sbin/httpd -DFOREGROUND
                  └─1853 /usr/sbin/httpd -DFOREGROUND

Feb 14 12:22:41 webmail1 systemd[1]: Starting The Apache HTTP Server...
Feb 14 12:22:43 webmail1 systemd[1]: Started The Apache HTTP Server.
# systemctl stop httpd.service 
..zzz...

The error_log instantaneously:
[Thu Feb 14 12:23:09.204078 2013] [mpm_prefork:notice] [pid 1847] AH00170: caught SIGWINCH, shutting down gracefully

In the meantime:
root      1752  1151  0 12:18 pts/0    00:00:00 tail -f /var/log/httpd/error_log
apache    1851     1  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    1852     1  0 12:22 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root      1857  1151  0 12:23 pts/0    00:00:00 systemctl stop httpd.service
root      1858  1857  0 12:23 pts/0    00:00:00 /usr/bin/systemd-tty-ask-password-agent --watch

Before you ask : yes there is SSL virtual host but the certificate is not password encrypted. Anyway why systemd-tty-ask-password-agent should ask for a password stopping httpd :-) ?

Comment 3 jean-filip tripcevic 2013-03-31 13:56:08 UTC
For what it's worth I have the same problem on latest F18 (Kernel 3.8.4).
I put the workaround in (KillMode=none) and problem goes away.


[Sun Mar 31 15:34:43.215636 2013] [mpm_prefork:notice] [pid 676] AH00170: caught SIGWINCH, shutting down gracefully
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0xb9306548 ***
======= Backtrace: =========
/lib/libc.so.6(+0x47b17ff9)[0xb7295ff9]
/lib/libapr-1.so.0(apr_pool_destroy+0x1ca)[0xb740d10a]
/etc/httpd/modules/mod_mpm_prefork.so(+0x27cf)[0xb70367cf]
/etc/httpd/modules/mod_mpm_prefork.so(+0x318b)[0xb703718b]
[0xb7705400]
/etc/httpd/modules/mod_authn_dbd.so(+0x9f9)[0xb71ac9f9]
/lib/ld-linux.so.2(+0x47a89d04)[0xb7715d04]
/lib/libc.so.6(+0x47ad3f01)[0xb7251f01]

Comment 4 Fedora Update System 2013-04-16 12:43:36 UTC
httpd-2.4.4-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/httpd-2.4.4-4.fc19

Comment 5 Fedora Update System 2013-04-16 16:12:11 UTC
Package httpd-2.4.4-4.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing httpd-2.4.4-4.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-5797/httpd-2.4.4-4.fc19
then log in and leave karma (feedback).

Comment 6 Fedora Update System 2013-04-25 13:55:12 UTC
httpd-2.4.4-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Fedora Update System 2013-05-17 12:41:35 UTC
httpd-2.4.4-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/httpd-2.4.4-3.fc18

Comment 8 Fedora Update System 2013-05-28 01:05:54 UTC
httpd-2.4.4-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.