Bug 117886

Summary: Apache 1.3.27 consumes 100% CPU
Product: Red Hat Enterprise Linux 3 Reporter: Franky Wong <fwong>
Component: httpdAssignee: Joe Orton <jorton>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 3.0CC: jparry
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2004-03-09 19:54:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Franky Wong 2004-03-09 17:34:40 UTC 
Description of problem:

I am using Apache 1.3.27 that a vendor provides, in perfork mode.
The server is receiving low traffic so it takes a few days for the 
problem to happen (unlike bug 117832, which is a high-usage server).

Given enough time, the httpd processes will consume all the CPU 
time.  The same problem happens to my httpd.worker threads (bug 
117832) and Xvnc (bug 117885) on another identical system.

Version-Release number of selected component (if applicable):
The Apache 1.3.27 is provided by a third-party vendor.

How reproducible:
Sometimes.

Steps to Reproduce:
1. Starts apache.
2. Wait for a few days.
  
Actual results: One of the apache processes starts to consume 100% 
CPU.  Given enough time, there will be enough processes consuming all 
CPU on the system.  They continue to consume CPU even when there is 
no http traffic to the system.

Expected results: All HTTP requests require little or no server 
processing.  Apache with the same configuration on other RHEL 2.1 
webservers, and serving the same HTTP requests, take up less than 10%
CPU time.

Additional info:

# uname -a
Linux madder 2.4.21-9.0.1.ELsmp #1 SMP Mon Feb 9 22:26:51 EST 2004 
i686 i686 i386 GNU/Linux

RHEL 3.0 was installed with all package collections.

I have these in /etc/sysctl.conf.

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
kernel.sysrq = 0
kernel.core_uses_pid = 1
fs.file-max = 65536
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_syncookies=1
net.ipv4.icmp_ignore_bogus_error_responses=1

Modules loaded in httpd.conf:

LoadModule ssl_module         libexec/libssl.so
LoadModule esi_module         libexec/mod_esi.so

SSL is loaded but there are no SSL requests to the system.  The "esi" 
module processes dynamic content.

# ps -fmHC httpd
UID        PID  PPID  C STIME TTY          TIME CMD
root      1762     1  0 Feb26 ?        00:00:23 /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls -R
nobody    2004  1762  0 Feb26 ?        01:05:15   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    2008  2004  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2019  2004  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2020  2004  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2021  2004  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2005  1762  0 Feb26 ?        01:05:15   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    2009  2005  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2010  2005  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2013  2005  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2024  2005  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2006  1762  0 Feb26 ?        01:05:10   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    2011  2006  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2012  2006  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2022  2006  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2023  2006  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2007  1762  0 Feb26 ?        01:06:03   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    2015  2007  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2025  2007  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2026  2007  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2027  2007  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2016  1762  0 Feb26 ?        01:05:15   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    2017  2016  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2018  2016  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2028  2016  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    2029  2016  0 Feb26 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8894  1762  0 Feb27 ?        01:05:39   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    8895  8894  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8896  8894  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8897  8894  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8898  8894  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8899  1762  0 Feb27 ?        00:00:01   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    8900  8899  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8901  8899  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8902  8899  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8903  8899  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8916  1762  0 Feb27 ?        01:05:27   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    8917  8916  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8918  8916  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8919  8916  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8920  8916  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8921  1762  0 Feb27 ?        00:00:01   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    8922  8921  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8923  8921  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8924  8921  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8925  8921  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8926  1762  0 Feb27 ?        01:05:09   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    8930  8926  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8931  8926  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8934  8926  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    8935  8926  0 Feb27 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9540  1762 14 01:47 ?        01:08:24   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9541  9540  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9542  9540  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9549  9540  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9550  9540  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9552  1762 14 01:47 ?        01:06:11   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9553  9552  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9554  9552  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9555  9552  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9556  9552  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9558  1762 13 01:47 ?        01:05:08   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9559  9558  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9560  9558  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9561  9558  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9562  9558  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9564  1762 14 01:47 ?        01:08:43   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9565  9564  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9566  9564  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9567  9564  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9568  9564  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9570  1762 13 01:47 ?        01:05:14   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9571  9570  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9572  9570  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9573  9570  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9574  9570  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9583  1762  0 01:47 ?        00:00:00   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9584  9583  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9585  9583  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9586  9583  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9587  9583  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9589  1762  0 01:47 ?        00:00:00   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9590  9589  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9591  9589  0 01:47 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9592  9589  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9593  9589  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9595  1762  0 01:48 ?        00:00:00   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9596  9595  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9597  9595  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9598  9595  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9599  9595  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9601  1762  0 01:48 ?        00:00:00   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9602  9601  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9603  9601  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9604  9601  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9605  9601  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9607  1762  0 01:48 ?        00:00:00   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody    9608  9607  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9609  9607  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9610  9607  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody    9611  9607  0 01:48 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody   10256  1762  0 08:55 ?        00:00:00   /raid/akamai/stage-
cls/bin/httpd -d /raid/akamai/stage-cls 
nobody   10257 10256  0 08:55 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody   10258 10256  0 08:55 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody   10259 10256  0 08:55 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl
nobody   10260 10256  0 08:55 ?        
00:00:00     /raid/akamai/stage-cls/bin/httpd -d /raid/akamai/stage-cl

# gdb
GNU gdb Red Hat Linux (6.0post-0.20031117.6rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and 
you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
This GDB was configured as "i386-redhat-linux-gnu".

(gdb) attach 9570
Attaching to process 9570
Reading symbols from /raid/akamai/stage-cls/bin/httpd...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Reading symbols from /lib/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread -1220291328 (LWP 9570)]
[New Thread -1273979984 (LWP 9574)]
[New Thread -1263490128 (LWP 9573)]
[New Thread -1245000784 (LWP 9572)]
[New Thread -1234510928 (LWP 9571)]
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /raid/akamai/stage-cls/libexec/mod_esi.so...done.
Loaded symbols for /raid/akamai/stage-cls/libexec/mod_esi.so
Reading symbols from /raid/akamai/stage-
cls/libexec/libicudt18l.so...done.
Loaded symbols for /raid/akamai/stage-cls/libexec/libicudt18l.so
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
0xb74b65e4 in strstr () from /lib/tls/libc.so.6

(gdb) info threads
  5 Thread -1234510928 (LWP 9571)  0xb75ebc32 in _dl_sysinfo_int80 () 
from /lib/ld-linux.so.2
  4 Thread -1245000784 (LWP 9572)  0xb75ebc32 in _dl_sysinfo_int80 () 
from /lib/ld-linux.so.2
  3 Thread -1263490128 (LWP 9573)  0xb75ebc32 in _dl_sysinfo_int80 () 
from /lib/ld-linux.so.2
  2 Thread -1273979984 (LWP 9574)  0xb75ebc32 in _dl_sysinfo_int80 () 
from /lib/ld-linux.so.2
  1 Thread -1220291328 (LWP 9570)  0xb74b65e4 in strstr () 
from /lib/tls/libc.so.6
(gdb) bt
#0  0xb74b65e4 in strstr () from /lib/tls/libc.so.6
#1  0xb6ab3cab in sendResponse () from /raid/akamai/stage-
cls/libexec/mod_esi.so
#2  0xb6abdfe5 in ap_proxy_http_handler () from /raid/akamai/stage-
cls/libexec/mod_esi.so
#3  0xb6ab01c6 in ?? () from /raid/akamai/stage-cls/libexec/mod_esi.so
#4  0x08086cf3 in ap_invoke_handler ()
#5  0x0809c089 in process_request_internal ()
#6  0x0809c0ec in ap_process_request ()
#7  0x08092fde in child_main ()
#8  0x0809323c in make_child ()
#9  0x080935b6 in perform_idle_server_maintenance ()
#10 0x08093b45 in standalone_main ()
#11 0x08094123 in main ()
Comment 1 Joe Orton 2004-03-09 19:54:25 UTC 
Sorry, we cannot help you with software provided by third parties;
please report problems in such software to the vendor who provided it
to you.