Bug 150295 - apache(libhttpd.ep) hangs on int_malloc
Summary: apache(libhttpd.ep) hangs on int_malloc
Alias: None
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: httpd   
(Show other bugs)
Version: 3.0
Hardware: athlon
OS: Linux
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Brian Brock
Depends On:
TreeView+ depends on / blocked
Reported: 2005-03-04 14:06 UTC by Rob Barker
Modified: 2008-04-09 18:29 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2005-05-03 17:41:08 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

Description Rob Barker 2005-03-04 14:06:40 UTC
Apache(1.3.33) process (libhttpd.ep) occasionally hangs on calling int_malloc. 
The hanging process usually consumes around 45% CPU.

An strace of the process returns no infomation.

A gdb backtrace reveals the following:

Heres a backtrace for one of the processes (the other one is almost identicle):

(gdb) backtrace
#0  0x009a9b26 in _int_malloc () from /lib/tls/libc.so.6
#1  0x009a8e9d in malloc () from /lib/tls/libc.so.6
#2  0x007516b6 in lmatcher () from /usr/local/apache_1.3.33/libexec/libhttpd.so
#3  0x00753616 in regexec () from /usr/local/apache_1.3.33/libexec/libhttpd.so
#4  0x0073fdef in ap_regexec () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#5  0x0070c25f in apply_rewrite_cond () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#6  0x0070b77c in apply_rewrite_rule () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#7  0x0070b20e in apply_rewrite_list () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#8  0x0070a131 in hook_uri2file () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#9  0x0072675e in run_method () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#10 0x007267d1 in ap_translate_name () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#11 0x0073d691 in process_request_internal () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#12 0x0073db71 in ap_process_request () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#13 0x00734352 in child_main () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#14 0x007345dd in make_child () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#15 0x00734958 in perform_idle_server_maintenance () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#16 0x00735020 in standalone_main () 
from /usr/local/apache_1.3.33/libexec/libhttpd.so
#17 0x007356ad in ap_main () from /usr/local/apache_1.3.33/libexec/libhttpd.so
#18 0x0804870b in main ()

Initially thought it due to apache being comiled against tls libraries instead 
of the usual ones in /lib.

but the problem still occured after recompiling agaisnt the non thread-safe 
libraries (from /lib). Again hangin on int_malloc.

At the point of failure there is approx 1.8g of memory available to the 

have tried exaclty the same apache/php(4.3.10) install on a redhat 7.3 
valhalla and apache has yet to produce the same error.

Can't reproduce the problem as it seems to only occure randomly (but usually 
when the server is under load).

Here's some server info:

[root@portal6 root]# uname -a
Linux portal6 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:50:31 EST 2004 i686 athlon 
i386 GNU/Linux
[root@portal6 root]# cat /etc/issue.net
Red Hat Enterprise Linux ES release 3 (Taroon Update 4)
Kernel \r on an \m
[root@portal6 root]# ldd /usr/local/apache/libexec/libhttpd.ep
        libhttpd.so => not found
        libm.so.6 => /lib/tls/libm.so.6 (0x00cbe000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x00a2b000)
        libgdbm.so.2 => /usr/lib/libgdbm.so.2 (0x00593000)
        libexpat.so.0 => /usr/local/lib/libexpat.so.0 (0x004d1000)
        libdl.so.2 => /lib/libdl.so.2 (0x00743000)
        libc.so.6 => /lib/tls/libc.so.6 (0x00d34000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x0082d000

Comment 1 Joe Orton 2005-03-04 15:37:16 UTC
This is probably some memory corruption issue, possibly in mod_rewrite, possibly
in the 1.3 code.  To track it down further, you could try a tool like valgrind
or ElectrictFence, or at least enable malloc checks in glibc; set:

  export MALLOC_CHECK_=2 /path/to/httpd

and libc will dump core if it detects heap corruption.  I'd suggest if possible
 not using the shared-code mode since it's so widely tested.

But regardless, since this is not a bug in a Red Hat distributed package, this
is not the place to report it.  You could report it upstream at
http://issues.apache.org/bugzilla/; if you can reproduce in a Red Hat-packaged
version of Apache, please reopen.

Comment 2 Rob Barker 2005-05-03 15:44:29 UTC
unfortunately had no information from the malloc check. 

Also tried running without Zend enabled in php and the error has not yet 
occured (after 3 weeks)

However we have tried using the apache2 RPM as supplied with this OS release - 

just got a rogue process with the same symptoms:

(gdb) backtrace
#0  0x00f2eb41 in _int_malloc () from /lib/tls/libc.so.6
#1  0x00f2de9d in malloc () from /lib/tls/libc.so.6
#2  0x010dcbdc in _emalloc (size=1) at /usr/local/src/php-
#3  0x010e92bb in zend_stack_init (stack=0x1163420) at /usr/local/src/php-
#4  0x010dd2b4 in zend_init_compiler_data_structures () at /usr/local/src/php-
#5  0x010dd372 in init_compiler () at /usr/local/src/php-
#6  0x010ea326 in zend_activate () at /usr/local/src/php-4.3.10/Zend/zend.c:636
#7  0x010c2c36 in php_request_startup () at /usr/local/src/php-
#8  0x010fd285 in php_apache_request_ctor (r=0x84b7cb0, ctx=0x8564850)
    at /usr/local/src/php-4.3.10/sapi/apache2handler/sapi_apache2.c:435
#9  0x010fd775 in php_handler (r=0x84b7cb0) at /usr/local/src/php-
#10 0x08068685 in ap_run_handler ()
#11 0x08068c9f in ap_invoke_handler ()
#12 0x08065326 in ap_process_request ()
#13 0x0806095c in _start ()
#14 0x084b7cb0 in ?? ()
#15 0x00000004 in ?? ()
#16 0x084b7cb0 in ?? ()
#17 0x0807228c in ap_run_pre_connection ()
#18 0x08072145 in ap_run_process_connection ()
#19 0x08066ba1 in ap_graceful_stop_signalled ()
#20 0x08066cf4 in ap_graceful_stop_signalled ()
#21 0x08066f99 in ap_graceful_stop_signalled ()
#22 0x08067630 in ap_mpm_run ()
#23 0x0806dacf in main ()

any thoughts?

Comment 3 Joe Orton 2005-05-03 17:41:08 UTC
Well, this one looks like a memory corruption issue in PHP 4.3.10 which you've
compiled yourself:

#0  0x00f2eb41 in _int_malloc () from /lib/tls/libc.so.6
#1  0x00f2de9d in malloc () from /lib/tls/libc.so.6
#2  0x010dcbdc in _emalloc (size=1) at /usr/local/src/php-
#3  0x010e92bb in zend_stack_init (stack=0x1163420) at /usr/local/src/php-

But again: there's no point in reporting these issues to us unless you are using
the PHP and httpd packages supplied by Red Hat.  We can't help you track this
issue down unless you do that.  If you can reproduce the problem using the
standard httpd and php packages, please re-open this bug.

Note You need to log in before you can comment on or make changes to this bug.