Bug 98056 - httpd child hogs 100% of 1 cpu
httpd child hogs 100% of 1 cpu
Status: CLOSED CURRENTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: httpd (Show other bugs)
9
All Linux
medium Severity medium
: ---
: ---
Assigned To: Joe Orton
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-06-25 16:37 EDT by Cott Lang
Modified: 2007-04-18 12:55 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-05-02 17:38:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Cott Lang 2003-06-25 16:37:50 EDT
Description of problem:

Randomly throughout the day, the httpd parent process will suddenly jump to hog
the entire CPU. -HUPing the process stops the problem. There are no log entries
in error.log that seem to be related. 

strace of the pid whether hogging or behaving normally just shows a
loop of:

select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
wait4(-1, 0xbfffea50, WNOHANG|WUNTRACED, NULL) = 0


Version-Release number of selected component (if applicable):

httpd-2.0.40-21.3

How reproducible:

It happens to me 2-3 times per day on each of 3 servers. I cannot force it to
occur, though.

Steps to Reproduce:
1. Do nothing
2. ...
3. BOOM!
    
Actual results:

CPU is hogged.  Occasionally, after several hours, the process seems to stop
hogging the CPU on it's own.

Expected results:

Normal CPU usage. :)

Additional info:

Happy to test stuff, ran out of ideas. :)
Comment 1 Joe Orton 2003-06-25 16:45:17 EDT
That's an strace of the httpd parent behaving normally.  Can you get an strace
of the process which is actually hogging CPU?

It's not just some child doing some heavy PHP script processing or something
like that?
Comment 2 Cott Lang 2003-06-25 20:02:25 EDT
Unfortunately, the trace is the same regardless of whether it's behaving
normally or hogging the cpu.  Non system calls ??

It's definitely the parent process - it's the only one running as root. 



Comment 3 Cott Lang 2003-06-30 11:45:04 EDT
Ok, my face is red.

It's not the parent process. The -HUP kills off the children, which solves the
problem.

top, whether in cummulative mode or not, only shows process time of the parent,
even though I'm in the default prefork mode (!).  I guess I'm just used to
Solaris top, but reading the man page makes it sound like it should NOT be doing
this?

ps -u apache -o 'pid pcpu' however, shows 1 or more children hogging all the CPU
they can grab.  strace -p child never returns anything. Killing the children
manually frees the cpu.  

Any ideas how I can get more information on this one ?

thanks :)
Comment 4 Joe Orton 2003-06-30 11:50:47 EDT
Can you post the results of:

$ gdb -p <childpid>

on a such a child?
Comment 5 Cott Lang 2003-06-30 15:56:11 EDT
[root@maggie log]# gdb -p 30420
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Attaching to process 30420
Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.
Reading symbols from /lib/libssl.so.4...(no debugging symbols found)...done.
Loaded symbols for /lib/libssl.so.4
Reading symbols from /lib/libcrypto.so.4...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2...
(no debugging symbols found)...done.
Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
Reading symbols from /usr/kerberos/lib/libkrb5.so.3...
(no debugging symbols found)...done.
Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
Reading symbols from /usr/kerberos/lib/libk5crypto.so.3...
(no debugging symbols found)...done.
Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
Reading symbols from /usr/kerberos/lib/libcom_err.so.3...
(no debugging symbols found)...done.
Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libaprutil.so.0...(no debugging symbols found)...
done.
Loaded symbols for /usr/lib/libaprutil.so.0
Reading symbols from /usr/lib/libgdbm.so.2...(no debugging symbols found)...
done.
Loaded symbols for /usr/lib/libgdbm.so.2
Reading symbols from /lib/libdb-4.0.so...done.
Loaded symbols for /lib/libdb-4.0.so
Reading symbols from /lib/tls/libpthread.so.0...done.
[New Thread 1077304384 (LWP 30420)]
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /usr/lib/libexpat.so.0...done.
Loaded symbols for /usr/lib/libexpat.so.0
Reading symbols from /usr/lib/libapr.so.0...done.
Loaded symbols for /usr/lib/libapr.so.0
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /etc/httpd/modules/mod_access.so...done.
Loaded symbols for /etc/httpd/modules/mod_access.so
Reading symbols from /etc/httpd/modules/mod_auth.so...done.
Loaded symbols for /etc/httpd/modules/mod_auth.so
Reading symbols from /etc/httpd/modules/mod_include.so...done.
Loaded symbols for /etc/httpd/modules/mod_include.so
Reading symbols from /etc/httpd/modules/mod_log_config.so...done.
Loaded symbols for /etc/httpd/modules/mod_log_config.so
Reading symbols from /etc/httpd/modules/mod_headers.so...done.
Loaded symbols for /etc/httpd/modules/mod_headers.so
Reading symbols from /etc/httpd/modules/mod_setenvif.so...done.
Loaded symbols for /etc/httpd/modules/mod_setenvif.so
Reading symbols from /etc/httpd/modules/mod_ssl.so...done.
Loaded symbols for /etc/httpd/modules/mod_ssl.so
Reading symbols from /etc/httpd/modules/mod_mime.so...done.
Loaded symbols for /etc/httpd/modules/mod_mime.so
Reading symbols from /etc/httpd/modules/mod_dir.so...done.
Loaded symbols for /etc/httpd/modules/mod_dir.so
Reading symbols from /etc/httpd/modules/mod_alias.so...done.
Loaded symbols for /etc/httpd/modules/mod_alias.so
Reading symbols from /etc/httpd/modules/mod_rewrite.so...done.
Loaded symbols for /etc/httpd/modules/mod_rewrite.so
Reading symbols from /etc/httpd/modules/mod_caucho.so...done.
Loaded symbols for /etc/httpd/modules/mod_caucho.so
0x08082b7a in pcre_compile ()
Comment 6 Joe Orton 2003-06-30 17:10:42 EDT
Sorry, can you get the backtrace out of gdb?  After gdb -p at the gdb prompt
enter "bt", and copy the backtrace here.
Comment 7 Cott Lang 2003-07-01 13:00:52 EDT
(gdb) bt
#0  0x08082b6e in pcre_compile ()
#1  0xbfffa960 in ?? ()
#2  0x0808494f in pcre_compile ()
#3  0x08084310 in pcre_compile ()
#4  0x0808384f in pcre_compile ()
#5  0x0808494f in pcre_compile ()
#6  0x08084310 in pcre_compile ()
#7  0x0808384f in pcre_compile ()
#8  0x0808494f in pcre_compile ()
#9  0x08084310 in pcre_compile ()
#10 0x0808384f in pcre_compile ()
#11 0x0808494f in pcre_compile ()
#12 0x08084310 in pcre_compile ()
#13 0x0808384f in pcre_compile ()
#14 0x0808494f in pcre_compile ()
#15 0x08084310 in pcre_compile ()
#16 0x0808384f in pcre_compile ()
#17 0x0808494f in pcre_compile ()
#18 0x08084310 in pcre_compile ()
#19 0x0808384f in pcre_compile ()
#20 0x0808494f in pcre_compile ()
#21 0x08084310 in pcre_compile ()
#22 0x0808384f in pcre_compile ()
#23 0x0808494f in pcre_compile ()
#24 0x08084310 in pcre_compile ()
#25 0x0808384f in pcre_compile ()
#26 0x0808494f in pcre_compile ()
#27 0x08084310 in pcre_compile ()
#28 0x0808384f in pcre_compile ()
#29 0x0808494f in pcre_compile ()
#30 0x08084310 in pcre_compile ()
#31 0x0808384f in pcre_compile ()
#32 0x0808494f in pcre_compile ()
#33 0x08084310 in pcre_compile ()
#34 0x0808384f in pcre_compile ()
#35 0x0808494f in pcre_compile ()
#36 0x08084310 in pcre_compile ()
#37 0x0808384f in pcre_compile ()
#38 0x0808494f in pcre_compile ()
#39 0x08084310 in pcre_compile ()
#40 0x0808384f in pcre_compile ()
#41 0x0808494f in pcre_compile ()
#42 0x08084310 in pcre_compile ()
#43 0x0808384f in pcre_compile ()
#44 0x0808494f in pcre_compile ()
#45 0x08084310 in pcre_compile ()
#46 0x0808384f in pcre_compile ()
#47 0x0808494f in pcre_compile ()
#48 0x08084310 in pcre_compile ()
#49 0x0808384f in pcre_compile ()
#50 0x0808494f in pcre_compile ()
---Type <return> to continue, or q <return> to quit---
#51 0x08084310 in pcre_compile ()
#52 0x0808384f in pcre_compile ()
#53 0x0808494f in pcre_compile ()
#54 0x08084310 in pcre_compile ()
#55 0x0808384f in pcre_compile ()
#56 0x0808494f in pcre_compile ()
#57 0x08084310 in pcre_compile ()
#58 0x0808384f in pcre_compile ()
#59 0x0808494f in pcre_compile ()
#60 0x080835b3 in pcre_compile ()
#61 0x0808494f in pcre_compile ()
#62 0x08082c06 in pcre_compile ()
#63 0x08084d23 in pcre_exec ()
#64 0x080800a4 in regexec ()
#65 0x403b20e4 in _init () from /etc/httpd/modules/mod_rewrite.so
#66 0x403b147b in _init () from /etc/httpd/modules/mod_rewrite.so
#67 0x403b101f in _init () from /etc/httpd/modules/mod_rewrite.so
#68 0x403aff9c in _init () from /etc/httpd/modules/mod_rewrite.so
#69 0x0807a885 in ap_run_translate_name ()
#70 0x0807b378 in ap_process_request_internal ()
#71 0x08062b78 in ap_process_request ()
#72 0x0805e25c in _start ()
#73 0x0806f345 in ap_run_process_connection ()
#74 0x080643f8 in ap_graceful_stop_signalled ()
#75 0x08064554 in ap_graceful_stop_signalled ()
#76 0x080647f9 in ap_graceful_stop_signalled ()
#77 0x08064e90 in ap_mpm_run ()
#78 0x0806af1f in main ()
#79 0x420156a4 in __libc_start_main () from /lib/tls/libc.so.6
Comment 8 Joe Orton 2003-07-10 11:20:59 EDT
The pcre_ symbols are suspect; the latest httpd and php packages have been
switched to using the system pcre library rather than each including a private
(and possibly conflicting) copy.
Comment 9 Allen Smith 2004-02-10 14:00:47 EST
I am seeing this on a RHEL3 box. httpd-2.0.46-26.ent Doesn't matter if
I run threaded or prefork. It happens whenever I hit a file in the
directory that has rewrite rules in the .htaccess. Only 4 lines of
rewrite rules.

Here is a partial backtrace from gdb.

#66 0x08087419 in pcre_compile ()
#67 0x08087f9f in pcre_compile ()
#68 0x08087960 in pcre_compile ()
#69 0x08087419 in pcre_compile ()
#70 0x08087f9f in pcre_compile ()
#71 0x08087e89 in pcre_compile ()
#72 0x08086256 in pcre_compile ()
#73 0x08088373 in pcre_exec ()
#74 0x08083724 in regexec ()
Comment 10 Joe Orton 2004-05-02 17:38:00 EDT
httpd and php have both been switched to using the system pcre library
for Fedora Core and RHEL3 updates, which is likely to be the cause of
this issue.  Please re-open if this is still reproducible in supported
releases.

Note You need to log in before you can comment on or make changes to this bug.