Bug 492753

Summary: Logrotate kills httpd after a "yum update"
Product: [Fedora] Fedora Reporter: Nadav Har'El <nyh>
Component: httpdAssignee: Joe Orton <jorton>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 10CC: benny+bugzilla, dnovotny, jbaker, jorton, pahan, simon.andrews, tsmetana
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-08-18 12:04:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nadav Har'El 2009-03-29 10:39:59 UTC
Today I realized that apache httpd that is always running on my machine, has mysteriously died. The last thing on the log was:

[Sun Mar 29 04:07:16 2009] [notice] SIGHUP received.  Attempting to restart
httpd: Syntax error on line 196 of /etc/httpd/conf/httpd.conf: Cannot load /etc/
httpd/modules/mod_proxy.so into server: /etc/httpd/modules/mod_proxy.so: undefin
ed symbol: ap_timeout_parameter_parse

After that, apache just died. So it was logrotate that killed my Apache using the /etc/logrotate.d/httpd script - which sends a SIGHUP to the running apache.

But why did SIGHUP kill my apache instead of just restarting? After some investigation, I figured out what happened, I think. What I am guessting happened is that sometime in the past, "yum updated" upgraded my httpd, and mod_proxy.so in particular, but didn't restart the server. When the (old) running httpd received a signal, it tried to reload the (new) modules, and that failed.

I'm not sure exactly what should be done to fix this problem. Should logrotate's script be changed to do something different (e.g., restart the httpd service completely, but contrast with bug 221073)? should "yum update" have done something different (like restart the httpd service immedately?) Should httpd have done something different (not reload a module which has already been loaded in the running process)?

But the end result is that, to make a long story short, logrotate killed my httpd. Which is why I am filing this bug for "logrotate".

Comment 1 Daniel Novotny 2009-03-30 07:15:03 UTC
signalling Apache to reset log files by SIGHUP is the right thing documented in Apache
(http://httpd.apache.org/docs/1.3/misc/howto.html#logreset)
so logrotate did nothing wrong. 

reassigning to httpd

Comment 2 Simon Andrews 2009-03-31 08:55:02 UTC
Just to add that our webserver had exactly the same problem following a recent logrotate:

[Sun Mar 29 04:14:10 2009] [notice] SIGHUP received.  Attempting to restart
httpd: Syntax error on line 188 of /etc/httpd/conf/httpd.conf: Cannot load /etc/httpd/modules/mod_proxy.so into server: /etc/httpd/modules/mod_proxy.so: undefined symbol: ap_timeout_parameter_parse

Manually restarting it brought apache back up without a complaint, but I can't see why it didn't work when logrotate tried to restart it.

Comment 3 Jon Baker 2009-06-03 21:06:31 UTC
We experienced the same issue as well.  

[Sun May 24 05:09:15 2009] [notice] SIGHUP received.  Attempting to restart
httpd: Syntax error on line 199 of /etc/httpd/conf/httpd.conf: Cannot load /etc/httpd/modules/mod_proxy.so into server: /etc/httpd/modules/mod_proxy.so: undefined symbol: ap_timeout_parameter_parse

Further, this is running x86_64 version, which may or may not be related.  I have 3 other servers running FC10's apache, one running x86_64, two running i386, but they have not experienced this problem, only this one (which is the one that receives probably 90% of the traffic load)

Comment 4 Jon Baker 2009-06-03 21:10:07 UTC
Versions of httpd in question:

Linux version 2.6.27.21-170.2.56.fc10.x86_64 - httpd-2.2.11-2.fc10.x86_64 - this is the server that failed

Linux version 2.6.27.23-78.2.50.fc9.i686 - httpd-2.2.11-2.fc10.i386 - running fine
 (this was updated to FC10 via yum, the fc10 kernel was older than the current fc9 kernel which is why it retained the fc9 kernel)

Linux version 2.6.27.23-78.2.50.fc9.i586 - httpd-2.2.11-2.fc10.i386 - running fine (same comment about the kernel as above)

Linux version 2.6.27.19-170.2.35.fc10.x86_64 - httpd-2.2.11-2.fc10.x86_64 - this is our x86_64 machine that is running fine.

Comment 5 Jon Baker 2009-06-08 15:07:40 UTC
Looks like an update has cured it for me, not sure which one but the logrotate script was re-written on an update (I had commented out the HUP line) and these are the modules that were updated at that time:

libX11-1.1.5-4.fc10.x86_64
netpbm-10.35.64-1.fc10.x86_64
policycoreutils-2.0.57-21.fc10.x86_64
perl-DBD-SQLite-1.23-1.fc10.x86_64
ipsec-tools-0.7.2-1.fc10.x86_64
3:ypbind-1.20.4-11.fc10.x86_64
selinux-policy-3.5.13-59.fc10.noarch
selinux-policy-targeted-3.5.13-59.fc10.noarch
xkeyboard-config-1.4-8.fc10.noarch
squirrelmail-1.4.18-2.fc10.noarch
libX11-1.1.5-4.fc10.i386
netpbm-10.35.64-1.fc10.i386
netpbm-progs-10.35.64-1.fc10.x86_64
libX11-devel-1.1.5-4.fc10.x86_64
libX11-devel-1.1.5-4.fc10.i386

Comment 6 Joe Orton 2009-08-18 12:04:21 UTC

*** This bug has been marked as a duplicate of bug 491567 ***