Bug 909703 - Apache does not work on apachectl graceful/restart or systemctl restart httpd.service
Summary: Apache does not work on apachectl graceful/restart or systemctl restart httpd...
Keywords:
Status: CLOSED DUPLICATE of bug 912288
Alias: None
Product: Fedora
Classification: Fedora
Component: httpd
Version: 18
Hardware: i686
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-02-10 15:46 UTC by Philippe Vouters
Modified: 2013-05-10 05:29 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-05-10 05:29:51 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
System Monitor Screenshot (199.21 KB, image/png)
2013-04-26 12:29 UTC, Philippe Vouters
no flags Details

Description Philippe Vouters 2013-02-10 15:46:05 UTC
Description of problem:
systemctl restart httpd.service hangs. Abortable via CTRL-C

Version-Release number of selected component (if applicable):
httpd-2.4.3-12.fc18.i686

How reproducible:
Always

Steps to Reproduce:
1. On a Fedora 18 x86 computer, yum install httpd-2.4.3-12.fc18.i686
2. systemctl start httpd.service
3. systemctl restart httpd.service
Alternative:
3. apachectl graceful
  
Actual results:

[philippe@victor ~]$ sudo systemctl restart httpd.service
^C[philippe@victor ~]$ ps -ef | grep httpd
apache    6267     1  0 16:16 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
philippe  6856 21252  0 16:29 pts/1    00:00:00 grep --color=auto httpd
root     20945 20872  0 08:59 pts/0    00:00:00 sudo tail -f /var/log/httpd/access_log
root     20947 20945  0 08:59 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo systemctl status httpd.service
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: deactivating (final-sigterm) since Sun 2013-02-10 16:28:15 CET; 1min 10s ago
         Process: 6820 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
        Main PID: 8547 (code=exited, status=0/SUCCESS)
          Status: "Total requests: 2555; Current requests/sec: 0; Current traffic:   0 B/sec"
          CGroup: name=systemd:/system/httpd.service
                  `-6267 /usr/sbin/httpd -DFOREGROUND

Feb 09 22:06:19 victor.vouters.dyndns.org httpd[8547]: [Sat Feb 09 22:06:19.0...
Feb 09 22:06:19 victor.vouters.dyndns.org httpd[8547]: [Sat Feb 09 22:06:19.0...
Feb 09 22:06:24 victor.vouters.dyndns.org systemd[1]: Started The Apache HTTP...
Feb 10 16:28:15 victor.vouters.dyndns.org systemd[1]: Stopping The Apache HTT...
Feb 10 16:28:16 victor.vouters.dyndns.org httpd[6820]: [Sun Feb 10 16:28:16.9...
Feb 10 16:28:16 victor.vouters.dyndns.org httpd[6820]: [Sun Feb 10 16:28:16.9...
[philippe@victor ~]$ ps -ef | grep httpd
apache    6267     1  0 16:16 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
philippe  6879 21252  0 16:29 pts/1    00:00:00 grep --color=auto httpd
root     20945 20872  0 08:59 pts/0    00:00:00 sudo tail -f /var/log/httpd/access_log
root     20947 20945  0 08:59 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo kill -TERM  httpd
[philippe@victor ~]$ ps -ef | grep httpd
philippe  6921 21252  0 16:30 pts/1    00:00:00 grep --color=auto httpd
root     20945 20872  0 08:59 pts/0    00:00:00 sudo tail -f /var/log/httpd/access_log
root     20947 20945  0 08:59 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo systemctl start httpd.service
[philippe@victor ~]$ ps -ef | grep httpd
root      6931     1  8 16:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    6934  6931  0 16:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    6935  6931  0 16:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    6936  6931  0 16:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    6937  6931  0 16:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    6938  6931  0 16:30 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
philippe  6941 21252  0 16:30 pts/1    00:00:00 grep --color=auto httpd
root     20945 20872  0 08:59 pts/0    00:00:00 sudo tail -f /var/log/httpd/access_log
root     20947 20945  0 08:59 pts/0    00:00:00 tail -f /var/log/httpd/access_log

Notice the CTRL-C on the first command and the apa&he process left existing.


Expected results:
Clean stop/start of the httpd daemon

Additional info:

Comment 1 customercare 2013-02-22 11:49:30 UTC
BTW: 

/usr/sbin/httpd -DFOREGROUND

this is also wrong, as demons do not start in foreground mode by default. foreground blocks the start process.

Comment 2 customercare 2013-02-22 12:10:27 UTC
fix it with a change of inside your unit file 

https://bugzilla.redhat.com/show_bug.cgi?id=907579

That will fork apache into background, like it always should do :)

Comment 3 Joe Orton 2013-02-22 13:09:32 UTC
"customercare" - you are providing workarounds rather than fixes.

Philippe, could you say:

a) What modules you have loaded?  (PHP etc)
b) Provide the output of "tail /var/log/httpd/error_log"

Comment 4 Philippe Vouters 2013-02-23 23:32:08 UTC
You have to correct /lib/systemd/system/httpd.service
which is NOT used with apachectl -k xxxxxx

[philippe@victor ~]$ sudo apachectl -k start
Passing arguments to httpd using apachectl is no longer supported.
You can only start/stop/restart httpd using this script.
If you want to pass extra arguments to httpd, edit the
/etc/sysconfig/httpd config file.
[philippe@victor ~]$ ps -ef | grep httpd
root      2468     1  4 00:24 ?        00:00:00 /usr/sbin/httpd -k start
apache    2471  2468  0 00:24 ?        00:00:00 /usr/sbin/httpd -k start
apache    2472  2468  0 00:24 ?        00:00:00 /usr/sbin/httpd -k start
apache    2473  2468  0 00:24 ?        00:00:00 /usr/sbin/httpd -k start
apache    2474  2468  0 00:24 ?        00:00:00 /usr/sbin/httpd -k start
apache    2475  2468  0 00:24 ?        00:00:00 /usr/sbin/httpd -k start
philippe  2478 29942  0 00:24 pts/1    00:00:00 grep --color=auto httpd
root     23238  1706  0 Feb23 pts/0    00:00:00 sudo tail -f /var/log/httpd/access_log
root     23240 23238  0 Feb23 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo apachectl -k stop 
Passing arguments to httpd using apachectl is no longer supported.
You can only start/stop/restart httpd using this script.
If you want to pass extra arguments to httpd, edit the
/etc/sysconfig/httpd config file.
[philippe@victor ~]$ ps -ef | grep http
philippe  2620 29942  0 00:28 pts/1    00:00:00 grep --color=auto http
root     23238  1706  0 Feb23 pts/0    00:00:00 sudo tail -f /var/log/httpd/access_log
root     23240 23238  0 Feb23 pts/0    00:00:00 tail -f /var/log/httpd/access_log
philippe 29917     1  0 Feb18 ?        00:00:00 /usr/libexec/gvfsd-http --spawner :1.3 /org/gtk/gvfs/exec_spaw/1

Comment 5 Philippe Vouters 2013-02-24 11:17:14 UTC
Here is what needs to be coded in /lib/systemd/system/httpd.service:

[philippe@victor ~]$ sudo cat /lib/systemd/system/httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -k start
#ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[philippe@victor ~]$

With this coding:

[philippe@victor ~]$ sudo systemctl --system daemon-reload
[philippe@victor ~]$ ps -ef | grep http
root      2034  1873  0 10:47 pts/1    00:00:00 sudo tail -f /var/log/httpd/access_log
root      2036  2034  0 10:47 pts/1    00:00:00 tail -f /var/log/httpd/access_log
philippe  5996  4234  0 12:05 pts/0    00:00:00 grep --color=auto http
[philippe@victor ~]$ sudo systemctl start httpd.service
[philippe@victor ~]$ ps -ef | grep http
root      2034  1873  0 10:47 pts/1    00:00:00 sudo tail -f /var/log/httpd/access_log
root      2036  2034  0 10:47 pts/1    00:00:00 tail -f /var/log/httpd/access_log
root      6004     1  7 12:05 ?        00:00:00 /usr/sbin/httpd -k start
apache    6006  6004  0 12:05 ?        00:00:00 /usr/sbin/httpd -k start
apache    6007  6004  0 12:05 ?        00:00:00 /usr/sbin/httpd -k start
apache    6008  6004  0 12:05 ?        00:00:00 /usr/sbin/httpd -k start
apache    6009  6004  0 12:05 ?        00:00:00 /usr/sbin/httpd -k start
apache    6010  6004  0 12:05 ?        00:00:00 /usr/sbin/httpd -k start
philippe  6013  4234  0 12:05 pts/0    00:00:00 grep --color=auto http
[philippe@victor ~]$ sudo systemctl stop httpd.service
[philippe@victor ~]$ ps -ef | grep http
root      2034  1873  0 10:47 pts/1    00:00:00 sudo tail -f /var/log/httpd/access_log
root      2036  2034  0 10:47 pts/1    00:00:00 tail -f /var/log/httpd/access_log
philippe  6185  4234  0 12:09 pts/0    00:00:00 grep --color=auto http
[philippe@victor ~]$ sudo systemctl stop httpd.service
[philippe@victor ~]$ ps -ef | grep http
root      2034  1873  0 10:47 pts/1    00:00:00 sudo tail -f /var/log/httpd/access_log
root      2036  2034  0 10:47 pts/1    00:00:00 tail -f /var/log/httpd/access_log
philippe  6185  4234  0 12:09 pts/0    00:00:00 grep --color=auto http
[philippe@victor ~]$ sudo systemctl restart httpd.service
[philippe@victor ~]$ ps -ef | grep http
root      2034  1873  0 10:47 pts/1    00:00:00 sudo tail -f /var/log/httpd/access_log
root      2036  2034  0 10:47 pts/1    00:00:00 tail -f /var/log/httpd/access_log
root      6227     1  4 12:10 ?        00:00:00 /usr/sbin/httpd -k start
apache    6230  6227  0 12:10 ?        00:00:00 /usr/sbin/httpd -k start
apache    6231  6227  0 12:10 ?        00:00:00 /usr/sbin/httpd -k start
apache    6232  6227  0 12:10 ?        00:00:00 /usr/sbin/httpd -k start
apache    6233  6227  0 12:10 ?        00:00:00 /usr/sbin/httpd -k start
apache    6234  6227  0 12:10 ?        00:00:00 /usr/sbin/httpd -k start
philippe  6237  4234  0 12:10 pts/0    00:00:00 grep --color=auto http
[philippe@victor ~]$ sudo systemctl status httpd.service
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: active (running) since Sun 2013-02-24 12:10:15 CET; 5min ago
         Process: 6223 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
         Process: 6226 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
        Main PID: 6227 (httpd)
          Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
          CGroup: name=systemd:/system/httpd.service
                  |-6227 /usr/sbin/httpd -k start
                  |-6230 /usr/sbin/httpd -k start
                  |-6231 /usr/sbin/httpd -k start
                  |-6232 /usr/sbin/httpd -k start
                  |-6233 /usr/sbin/httpd -k start
                  `-6234 /usr/sbin/httpd -k start

Feb 24 12:10:13 victor.vouters.dyndns.org systemd[1]: Starting The Apache HTT...
Feb 24 12:10:15 victor.vouters.dyndns.org systemd[1]: Started The Apache HTTP...
[philippe@victor ~]$

Comment 6 Philippe Vouters 2013-02-26 17:01:28 UTC
Dear Customercare,

My only way to get $ sudo apachectl {start|stop|restart|graceful| and so on} work like before as well as $ sudo systemctl {start|stop|restart} httpd.service has been to modify /lib/systemd/system/httpd.service this way

[philippe@victor C]$ sudo cat /lib/systemd/system/httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -k start
#ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[philippe@victor C]$

This is exactly what I document in the corresponding Red Hat's bugzilla report.

Grüsse von Frankreich

Comment 7 Philippe Vouters 2013-04-25 17:10:12 UTC
Dear Red Hat httpd package maintainer,

I want and need to inform you that my change to /lib/systemd/system/httpd.service although seemingly working is not perfect. I have not correctly isolated the actual problem.

One direction I am inclined to look at is that Apache 2.4 would create over the time additional child processes (at least one in my case) which would be out of the knowledge of systemctl (so out of its control).

Stay tuned awaiting for more information.

Best regards,
Philippe Vouters (Fontainebleau/France)

Comment 8 Philippe Vouters 2013-04-26 11:08:50 UTC
Dear Red Hat httpd package maintainer,

The direction where I was inclined to look at (read previous post) which could explain why sudo systemctl stop httpd.service could hang does not look obvious. I do notice that unlike what I was usually doing my two actions are stop followed by start instead of a single restart.

[philippe@victor ~]$ ps -ef | grep "/usr/sbin/httpd -k start" | wc -l
8
[philippe@victor ~]$ sudo systemctl status httpd.service | grep "/usr/sbin/httpd -k start" | wc -l
7
[philippe@victor ~]$ sudo systemctl stop httpd.service                          [philippe@victor ~]$ sudo systemctl start httpd.service                         [philippe@victor ~]$ 

Stay tuned for more input.
Best regards,
Philippe Vouters (Fontainebleau/France)

Comment 9 Philippe Vouters 2013-04-26 11:56:47 UTC
Rather looks like there is a race condition within systemctl with it not being reliable enough. So far I have only been able to notice it with the Apache service.

[philippe@victor ~]$ sudo vi /etc/httpd/conf.d/tima.conf
[philippe@victor ~]$ sudo systemctl restart httpd.service                       [philippe@victor ~]$ sudo vi /etc/sysconfig/iptables
[philippe@victor ~]$ sudo systemctl restart httpd.service (HANGED)
^C
[philippe@victor ~]ps -ef | grep "/usr/sbin/httpd -k start" | wc -l
2
[philippe@victor ~]$ ps -ef | grep "/usr/sbin/httpd -k start"                   apache   26458     1  0 13:19 ?        00:00:00 /usr/sbin/httpd -k start
philippe 26643 23574  0 13:23 pts/1    00:00:00 grep --color=auto /usr/sbin/httpd -k start
[philippe@victor ~]$ sudo systemctl status httpd.service |  grep "/usr/sbin/httpd -k start"
                  26656 /usr/sbin/httpd -k start 
                  26658 /usr/sbin/httpd -k start 
                  26659 /usr/sbin/httpd -k start 
                  26660 /usr/sbin/httpd -k start 
                  26661 /usr/sbin/httpd -k start 
                  26662 /usr/sbin/httpd -k start 
[philippe@victor ~]$ sudo systemctl restart iptables.service
[philippe@victor ~]$ sudo systemctl status httpd.service
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: active (running) since Fri 2013-04-26 13:23:39 CEST; 8min ago
         Process: 26583 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
         Process: 26654 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
        Main PID: 26656 (/usr/sbin/httpd)
          CGroup: name=systemd:/system/httpd.service
                  26656 /usr/sbin/httpd -k start
                  26658 /usr/sbin/httpd -k start
                  26659 /usr/sbin/httpd -k start
                  26660 /usr/sbin/httpd -k start
                  26661 /usr/sbin/httpd -k start
                  26662 /usr/sbin/httpd -k start
                  26926 /usr/sbin/httpd -k start

Apr 26 13:23:38 victor.vouters.dyndns.org systemd[1]: Starting The Apache HTT...
Apr 26 13:23:39 victor.vouters.dyndns.org systemd[1]: Started The Apache HTTP...
[philippe@victor ~]$ 

Interestingly and despite the fact that, unlike previously, I did not take any action to manually kill -9 the remaining Apache process, I now read:

[philippe@victor ~]$ ps -ef | grep "/usr/sbin/httpd -k start"                   root     26656     1  0 13:23 ?        00:00:00 /usr/sbin/httpd -k start
apache   26658 26656  0 13:23 ?        00:00:00 /usr/sbin/httpd -k start
apache   26659 26656  0 13:23 ?        00:00:00 /usr/sbin/httpd -k start
apache   26660 26656  0 13:23 ?        00:00:00 /usr/sbin/httpd -k start
apache   26661 26656  0 13:23 ?        00:00:00 /usr/sbin/httpd -k start
apache   26662 26656  0 13:23 ?        00:00:00 /usr/sbin/httpd -k start
apache   26926 26656  0 13:27 ?        00:00:00 /usr/sbin/httpd -k start
apache   27481 26656  0 13:39 ?        00:00:00 /usr/sbin/httpd -k start
philippe 27569 23574  0 13:41 pts/1    00:00:00 grep --color=auto /usr/sbin/httpd -k start

AND THE SURVIVING APACHE PROCESS DISAPPEARED:

[philippe@victor ~]$ ps -ef | grep "/usr/sbin/httpd -k start" |  grep 26458
[philippe@victor ~]$ 

So next time I shall NOT CRTL-C '$ sudo systemctl restart httpd.service' if it appears to hang and observe things from another terminal. The actual problem definition may be something totally other. A likelyhood can be Apache too slow to graceful-stop.

Stay tuned.
Warmest regards,
Philippe Vouters (Fontainebleau/France)

Comment 10 Philippe Vouters 2013-04-26 12:29:51 UTC
Created attachment 740381 [details]
System Monitor Screenshot

Comment 11 Philippe Vouters 2013-04-26 12:59:17 UTC
Oops ! I lost what I entered. The truth I now come up with is that $ sudo systemctl restart httpd.service appears to hang but it does NOT. The software action reveals abnormally terribly slow with no visible reason in CPU/Memory/Swap/Network activity (see System Monitor Screenshot).

I document http://vouters.dyndns.org/tima/All-OS-Apache-Invalidating_Apache_server_cache.html on my Web site with the htcacheclean action I took at Apache 2.2 times. I can't view my disk activity (the disk LED does NOT work on my desktop as well as my disk making no noise as well as it being a MAXTOR SATA disk with a memory backup cache proving far much faster than my old ATA Seagate disk).

My next step will thus be to redo the htcacheclean action or, better, shall dig over Apache 2.4 Web help to completely invalidate any Apache disk cache. Will it improve the systemctl restart httpd.service response time ?

Another possible direction where I can look at is a Apache lock not correctly and fast enough unlocked.

Stay tuned.
Philippe

Comment 12 Philippe Vouters 2013-04-26 13:19:55 UTC
Did the following change inside my Web site Apache configuration file: The more so as I read:

[philippe@victor ~]$ sudo lsof | grep httpd | more | grep mod_cache
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/500/gvfs
      Output information may be incomplete.
/usr/sbin 31293           root  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31293           root  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
/usr/sbin 31295         apache  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31295         apache  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
/usr/sbin 31296         apache  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31296         apache  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
/usr/sbin 31297         apache  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31297         apache  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
/usr/sbin 31298         apache  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31298         apache  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
/usr/sbin 31299         apache  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31299         apache  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
/usr/sbin 31459         apache  mem       REG        8,4     31312    4559390 /usr/lib/httpd/modules/mod_cache_disk.so
/usr/sbin 31459         apache  mem       REG        8,4     64256    4555408 /usr/lib/httpd/modules/mod_cache.so
[philippe@victor ~]$ 

Here my unique Web site Apache configuration change:

#
# Disable the cache lock
#
<IfModule mod_cache.c>
    CacheLock off
    CacheLockPath /tmp/mod_cache-lock
    CacheLockMaxAge 5
</IfModule>

This is suggested by http://httpd.apache.org/docs/current/mod/mod_cache.html
I tried several successive sudo systemctl restart httpd.service and could not incur any abnormal long delay to return to the bash prompt.

To be further confirmed over time.

Stay tuned.
Philippe

Comment 13 Philippe Vouters 2013-04-26 14:40:31 UTC
Looks there is a bug inside httpd or in libc (the int_free call retrieved twice on the call stack). httpd is linked with libpthread.so.0. A likelyhood is that int_free (libc calls in general ???) is not signal-safe. This is showed by this fact on the gdb backtrace below: "#8  <signal handler called>" and the call trace which preceedes. This reminds me my fix proposal to aterm (https://bugzilla.redhat.com/show_bug.cgi?id=835908). If my fix proposal to aterm is what should be done, the bug is in Apache 2.4 forgetting to protect itself against some signal(s) to be determined.

During a hanging systemctl restart and the knowledge of the httpd pid,
I have been doing:

[philippe@victor ~]$ sudo strace -v -e trace=all -f -o apache.log -p 1048
[philippe@victor ~]$ sudo cat apache.log
1048  futex(0xb73a4420, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
1048  +++ killed by SIGKILL +++

and
$ sudo gdb /usr/sbin/httpd -p 1558
(gdb) where
#0  0xb76bf424 in __kernel_vsyscall ()
#1  0xb72b67d2 in __lll_lock_wait_private () from /lib/libc.so.6
#2  0xb722de93 in _L_lock_4410 () from /lib/libc.so.6
#3  0xb7228d79 in _int_free () from /lib/libc.so.6
#4  0xb739f7ae in apr_allocator_destroy () from /lib/libapr-1.so.0
#5  0xb73a015c in apr_pool_destroy () from /lib/libapr-1.so.0
#6  0xb6ff17cf in clean_child_exit ()
   from /etc/httpd/modules/mod_mpm_prefork.so
#7  0xb6ff218b in just_die () from /etc/httpd/modules/mod_mpm_prefork.so
#8  <signal handler called>
#9  0xb7228b7f in _int_free () from /lib/libc.so.6
#10 0xb68f6456 in zend_hash_destroy () from /etc/httpd/modules/libphp5.so
#11 0xb68ddd50 in destroy_zend_class () from /etc/httpd/modules/libphp5.so
#12 0xb68f643e in zend_hash_destroy () from /etc/httpd/modules/libphp5.so
#13 0xb68e8147 in zend_shutdown () from /etc/httpd/modules/libphp5.so
#14 0xb6883262 in php_module_shutdown () from /etc/httpd/modules/libphp5.so
#15 0xb6883345 in php_module_shutdown_wrapper ()
   from /etc/httpd/modules/libphp5.so
#16 0xb699a18f in php_apache_child_shutdown ()
   from /etc/httpd/modules/libphp5.so
#17 0xb739ffbe in apr_pool_destroy () from /lib/libapr-1.so.0
#18 0xb6ff17cf in clean_child_exit ()
   from /etc/httpd/modules/mod_mpm_prefork.so

before the process gets killed.

Comment 14 Jan Kaluža 2013-04-29 07:50:33 UTC
This last crash looks like this fixed Bug 912288 fixed in httpd-2.4.4-4.fc19.

Comment 15 Philippe Vouters 2013-04-29 14:17:58 UTC
Unless I misunderstood, this fix ought to be in systemctl.
"fix service file to not send SIGTERM after ExecStop (#906321, #912288)"
This would mean other services might also be impacted. I would need to test this specific fix built for FC18 i686. How can I manage to get this accurate fix WITHOUT modifying ANY other code parts ?

This way I shall be able to best confirm this SIGTERM sending by systemctl is the actual root cause of the problems I encountered. Perhaps the other FC18 X86_64 user who reported a heap httpd glibc crash on systemctl stop httpd.service may also confirm.

Also I notice that /sbin/apachectl is very bound to /usr/bin/systemctl activations (start|stop|restart|status|graceful|graceful-stop).

Comment 16 Philippe Vouters 2013-04-29 19:24:59 UTC
Provided systemctl was allowing kill -TERM|HUP|<other> `cat /var/run/pid` there would be less problems.

[philippe@victor ~] # these httpd processes have started manually with 
[philippe@victor ~] # sudo httpd -k start,
[philippe@victor ~]$ ps -ef | grep httpd
philippe  9362 23363  0 21:07 pts/1    00:00:00 grep --color=auto httpd
philippe 10480  1775  0 Apr28 pts/0    00:00:00 tail -f /var/log/httpd/access_log
root     12498     1  0 14:02 ?        00:00:00 /sbin/httpd -k start
apache   12501 12498  0 14:02 ?        00:00:00 /sbin/httpd -k start
apache   12502 12498  0 14:02 ?        00:00:00 /sbin/httpd -k start
apache   12503 12498  0 14:02 ?        00:00:00 /sbin/httpd -k start
apache   12504 12498  0 14:02 ?        00:00:00 /sbin/httpd -k start
apache   12505 12498  0 14:02 ?        00:00:00 /sbin/httpd -k start
apache   12707 12498  0 14:07 ?        00:00:00 /sbin/httpd -k start
apache   15291 12498  0 15:13 ?        00:00:00 /sbin/httpd -k start
apache   15621 12498  0 15:22 ?        00:00:00 /sbin/httpd -k start
apache   15623 12498  0 15:22 ?        00:00:00 /sbin/httpd -k start
apache   15624 12498  0 15:22 ?        00:00:00 /sbin/httpd -k start
[philippe@victor ~]$ sudo kill -TERM 12498
[philippe@victor ~]$ ps -ef | grep httpd
philippe  9378 23363  0 21:07 pts/1    00:00:00 grep --color=auto httpd
philippe 10480  1775  0 Apr28 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo systemctl start httpd.service
[philippe@victor ~]$ ps -ef | grep httpd
root      9403     1  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
apache    9406  9403  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
apache    9407  9403  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
apache    9408  9403  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
apache    9409  9403  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
apache    9410  9403  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
apache    9411  9403  0 21:08 ?        00:00:00 /usr/sbin/httpd -k start
philippe  9609 23363  0 21:10 pts/1    00:00:00 grep --color=auto httpd
philippe 10480  1775  0 Apr28 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo kill -TERM 9403
[philippe@victor ~]$ ps -ef | grep httpd
philippe  9624 23363  0 21:10 pts/1    00:00:00 grep --color=auto httpd
philippe 10480  1775  0 Apr28 pts/0    00:00:00 tail -f /var/log/httpd/access_log
[philippe@victor ~]$ sudo systemctl status httpd.service
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: inactive (dead) since Mon 2013-04-29 21:10:44 CEST; 20s ago
         Process: 9619 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
         Process: 9401 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
        Main PID: 9403 (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/httpd.service

Apr 29 21:08:14 victor.vouters.dyndns.org systemd[1]: Started The Apache HTTP...
Apr 29 21:10:44 victor.vouters.dyndns.org httpd[9619]: httpd (no pid file) no...
[philippe@victor ~]$ sudo systemctl start httpd.service
[philippe@victor ~]$ sudo systemctl status httpd.service
httpd.service - The Apache HTTP Server
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
          Active: active (running) since Mon 2013-04-29 21:11:50 CEST; 26s ago
         Process: 9619 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS)
         Process: 9669 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
        Main PID: 9670 (/usr/sbin/httpd)
          CGroup: name=systemd:/system/httpd.service
                  9670 /usr/sbin/httpd -k start
                  9673 /usr/sbin/httpd -k start
                  9674 /usr/sbin/httpd -k start
                  9675 /usr/sbin/httpd -k start
                  9676 /usr/sbin/httpd -k start
                  9677 /usr/sbin/httpd -k start

Apr 29 21:11:49 victor.vouters.dyndns.org systemd[1]: Starting The Apache HTT...
Apr 29 21:11:50 victor.vouters.dyndns.org systemd[1]: Started The Apache HTTP...

Comment 17 Jan Kaluža 2013-05-10 05:29:51 UTC
Well, the fix was confirmed in Bug 912288 and packages are already in F18 repository. The symptoms and also the backtrace are the same as in mentioned bug, so I'm closing this as duplicate. If you find out your problem persists, feel free to reopen.

*** This bug has been marked as a duplicate of bug 912288 ***


Note You need to log in before you can comment on or make changes to this bug.