Description of problem: Randomly throughout the day, the httpd parent process will suddenly jump to hog the entire CPU. -HUPing the process stops the problem. There are no log entries in error.log that seem to be related. strace of the pid whether hogging or behaving normally just shows a loop of: select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) wait4(-1, 0xbfffea50, WNOHANG|WUNTRACED, NULL) = 0 Version-Release number of selected component (if applicable): httpd-2.0.40-21.3 How reproducible: It happens to me 2-3 times per day on each of 3 servers. I cannot force it to occur, though. Steps to Reproduce: 1. Do nothing 2. ... 3. BOOM! Actual results: CPU is hogged. Occasionally, after several hours, the process seems to stop hogging the CPU on it's own. Expected results: Normal CPU usage. :) Additional info: Happy to test stuff, ran out of ideas. :)
That's an strace of the httpd parent behaving normally. Can you get an strace of the process which is actually hogging CPU? It's not just some child doing some heavy PHP script processing or something like that?
Unfortunately, the trace is the same regardless of whether it's behaving normally or hogging the cpu. Non system calls ?? It's definitely the parent process - it's the only one running as root.
Ok, my face is red. It's not the parent process. The -HUP kills off the children, which solves the problem. top, whether in cummulative mode or not, only shows process time of the parent, even though I'm in the default prefork mode (!). I guess I'm just used to Solaris top, but reading the man page makes it sound like it should NOT be doing this? ps -u apache -o 'pid pcpu' however, shows 1 or more children hogging all the CPU they can grab. strace -p child never returns anything. Killing the children manually frees the cpu. Any ideas how I can get more information on this one ? thanks :)
Can you post the results of: $ gdb -p <childpid> on a such a child?
[root@maggie log]# gdb -p 30420 GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". Attaching to process 30420 Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done. Reading symbols from /lib/libssl.so.4...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.4 Reading symbols from /lib/libcrypto.so.4...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.4 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2... (no debugging symbols found)...done. Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2 Reading symbols from /usr/kerberos/lib/libkrb5.so.3... (no debugging symbols found)...done. Loaded symbols for /usr/kerberos/lib/libkrb5.so.3 Reading symbols from /usr/kerberos/lib/libk5crypto.so.3... (no debugging symbols found)...done. Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3 Reading symbols from /usr/kerberos/lib/libcom_err.so.3... (no debugging symbols found)...done. Loaded symbols for /usr/kerberos/lib/libcom_err.so.3 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/libaprutil.so.0...(no debugging symbols found)... done. Loaded symbols for /usr/lib/libaprutil.so.0 Reading symbols from /usr/lib/libgdbm.so.2...(no debugging symbols found)... done. Loaded symbols for /usr/lib/libgdbm.so.2 Reading symbols from /lib/libdb-4.0.so...done. Loaded symbols for /lib/libdb-4.0.so Reading symbols from /lib/tls/libpthread.so.0...done. [New Thread 1077304384 (LWP 30420)] Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /usr/lib/libexpat.so.0...done. Loaded symbols for /usr/lib/libexpat.so.0 Reading symbols from /usr/lib/libapr.so.0...done. Loaded symbols for /usr/lib/libapr.so.0 Reading symbols from /lib/tls/libm.so.6...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libnsl.so.1...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /etc/httpd/modules/mod_access.so...done. Loaded symbols for /etc/httpd/modules/mod_access.so Reading symbols from /etc/httpd/modules/mod_auth.so...done. Loaded symbols for /etc/httpd/modules/mod_auth.so Reading symbols from /etc/httpd/modules/mod_include.so...done. Loaded symbols for /etc/httpd/modules/mod_include.so Reading symbols from /etc/httpd/modules/mod_log_config.so...done. Loaded symbols for /etc/httpd/modules/mod_log_config.so Reading symbols from /etc/httpd/modules/mod_headers.so...done. Loaded symbols for /etc/httpd/modules/mod_headers.so Reading symbols from /etc/httpd/modules/mod_setenvif.so...done. Loaded symbols for /etc/httpd/modules/mod_setenvif.so Reading symbols from /etc/httpd/modules/mod_ssl.so...done. Loaded symbols for /etc/httpd/modules/mod_ssl.so Reading symbols from /etc/httpd/modules/mod_mime.so...done. Loaded symbols for /etc/httpd/modules/mod_mime.so Reading symbols from /etc/httpd/modules/mod_dir.so...done. Loaded symbols for /etc/httpd/modules/mod_dir.so Reading symbols from /etc/httpd/modules/mod_alias.so...done. Loaded symbols for /etc/httpd/modules/mod_alias.so Reading symbols from /etc/httpd/modules/mod_rewrite.so...done. Loaded symbols for /etc/httpd/modules/mod_rewrite.so Reading symbols from /etc/httpd/modules/mod_caucho.so...done. Loaded symbols for /etc/httpd/modules/mod_caucho.so 0x08082b7a in pcre_compile ()
Sorry, can you get the backtrace out of gdb? After gdb -p at the gdb prompt enter "bt", and copy the backtrace here.
(gdb) bt #0 0x08082b6e in pcre_compile () #1 0xbfffa960 in ?? () #2 0x0808494f in pcre_compile () #3 0x08084310 in pcre_compile () #4 0x0808384f in pcre_compile () #5 0x0808494f in pcre_compile () #6 0x08084310 in pcre_compile () #7 0x0808384f in pcre_compile () #8 0x0808494f in pcre_compile () #9 0x08084310 in pcre_compile () #10 0x0808384f in pcre_compile () #11 0x0808494f in pcre_compile () #12 0x08084310 in pcre_compile () #13 0x0808384f in pcre_compile () #14 0x0808494f in pcre_compile () #15 0x08084310 in pcre_compile () #16 0x0808384f in pcre_compile () #17 0x0808494f in pcre_compile () #18 0x08084310 in pcre_compile () #19 0x0808384f in pcre_compile () #20 0x0808494f in pcre_compile () #21 0x08084310 in pcre_compile () #22 0x0808384f in pcre_compile () #23 0x0808494f in pcre_compile () #24 0x08084310 in pcre_compile () #25 0x0808384f in pcre_compile () #26 0x0808494f in pcre_compile () #27 0x08084310 in pcre_compile () #28 0x0808384f in pcre_compile () #29 0x0808494f in pcre_compile () #30 0x08084310 in pcre_compile () #31 0x0808384f in pcre_compile () #32 0x0808494f in pcre_compile () #33 0x08084310 in pcre_compile () #34 0x0808384f in pcre_compile () #35 0x0808494f in pcre_compile () #36 0x08084310 in pcre_compile () #37 0x0808384f in pcre_compile () #38 0x0808494f in pcre_compile () #39 0x08084310 in pcre_compile () #40 0x0808384f in pcre_compile () #41 0x0808494f in pcre_compile () #42 0x08084310 in pcre_compile () #43 0x0808384f in pcre_compile () #44 0x0808494f in pcre_compile () #45 0x08084310 in pcre_compile () #46 0x0808384f in pcre_compile () #47 0x0808494f in pcre_compile () #48 0x08084310 in pcre_compile () #49 0x0808384f in pcre_compile () #50 0x0808494f in pcre_compile () ---Type <return> to continue, or q <return> to quit--- #51 0x08084310 in pcre_compile () #52 0x0808384f in pcre_compile () #53 0x0808494f in pcre_compile () #54 0x08084310 in pcre_compile () #55 0x0808384f in pcre_compile () #56 0x0808494f in pcre_compile () #57 0x08084310 in pcre_compile () #58 0x0808384f in pcre_compile () #59 0x0808494f in pcre_compile () #60 0x080835b3 in pcre_compile () #61 0x0808494f in pcre_compile () #62 0x08082c06 in pcre_compile () #63 0x08084d23 in pcre_exec () #64 0x080800a4 in regexec () #65 0x403b20e4 in _init () from /etc/httpd/modules/mod_rewrite.so #66 0x403b147b in _init () from /etc/httpd/modules/mod_rewrite.so #67 0x403b101f in _init () from /etc/httpd/modules/mod_rewrite.so #68 0x403aff9c in _init () from /etc/httpd/modules/mod_rewrite.so #69 0x0807a885 in ap_run_translate_name () #70 0x0807b378 in ap_process_request_internal () #71 0x08062b78 in ap_process_request () #72 0x0805e25c in _start () #73 0x0806f345 in ap_run_process_connection () #74 0x080643f8 in ap_graceful_stop_signalled () #75 0x08064554 in ap_graceful_stop_signalled () #76 0x080647f9 in ap_graceful_stop_signalled () #77 0x08064e90 in ap_mpm_run () #78 0x0806af1f in main () #79 0x420156a4 in __libc_start_main () from /lib/tls/libc.so.6
The pcre_ symbols are suspect; the latest httpd and php packages have been switched to using the system pcre library rather than each including a private (and possibly conflicting) copy.
I am seeing this on a RHEL3 box. httpd-2.0.46-26.ent Doesn't matter if I run threaded or prefork. It happens whenever I hit a file in the directory that has rewrite rules in the .htaccess. Only 4 lines of rewrite rules. Here is a partial backtrace from gdb. #66 0x08087419 in pcre_compile () #67 0x08087f9f in pcre_compile () #68 0x08087960 in pcre_compile () #69 0x08087419 in pcre_compile () #70 0x08087f9f in pcre_compile () #71 0x08087e89 in pcre_compile () #72 0x08086256 in pcre_compile () #73 0x08088373 in pcre_exec () #74 0x08083724 in regexec ()
httpd and php have both been switched to using the system pcre library for Fedora Core and RHEL3 updates, which is likely to be the cause of this issue. Please re-open if this is still reproducible in supported releases.