Red Hat Bugzilla – Bug 150295
apache(libhttpd.ep) hangs on int_malloc
Last modified: 2008-04-09 14:29:28 EDT
Apache(1.3.33) process (libhttpd.ep) occasionally hangs on calling int_malloc. The hanging process usually consumes around 45% CPU. An strace of the process returns no infomation. A gdb backtrace reveals the following: Heres a backtrace for one of the processes (the other one is almost identicle): (gdb) backtrace #0 0x009a9b26 in _int_malloc () from /lib/tls/libc.so.6 #1 0x009a8e9d in malloc () from /lib/tls/libc.so.6 #2 0x007516b6 in lmatcher () from /usr/local/apache_1.3.33/libexec/libhttpd.so #3 0x00753616 in regexec () from /usr/local/apache_1.3.33/libexec/libhttpd.so #4 0x0073fdef in ap_regexec () from /usr/local/apache_1.3.33/libexec/libhttpd.so #5 0x0070c25f in apply_rewrite_cond () from /usr/local/apache_1.3.33/libexec/libhttpd.so #6 0x0070b77c in apply_rewrite_rule () from /usr/local/apache_1.3.33/libexec/libhttpd.so #7 0x0070b20e in apply_rewrite_list () from /usr/local/apache_1.3.33/libexec/libhttpd.so #8 0x0070a131 in hook_uri2file () from /usr/local/apache_1.3.33/libexec/libhttpd.so #9 0x0072675e in run_method () from /usr/local/apache_1.3.33/libexec/libhttpd.so #10 0x007267d1 in ap_translate_name () from /usr/local/apache_1.3.33/libexec/libhttpd.so #11 0x0073d691 in process_request_internal () from /usr/local/apache_1.3.33/libexec/libhttpd.so #12 0x0073db71 in ap_process_request () from /usr/local/apache_1.3.33/libexec/libhttpd.so #13 0x00734352 in child_main () from /usr/local/apache_1.3.33/libexec/libhttpd.so #14 0x007345dd in make_child () from /usr/local/apache_1.3.33/libexec/libhttpd.so #15 0x00734958 in perform_idle_server_maintenance () from /usr/local/apache_1.3.33/libexec/libhttpd.so #16 0x00735020 in standalone_main () from /usr/local/apache_1.3.33/libexec/libhttpd.so #17 0x007356ad in ap_main () from /usr/local/apache_1.3.33/libexec/libhttpd.so #18 0x0804870b in main () Initially thought it due to apache being comiled against tls libraries instead of the usual ones in /lib. but the problem still occured after recompiling agaisnt the non thread-safe libraries (from /lib). Again hangin on int_malloc. At the point of failure there is approx 1.8g of memory available to the application. have tried exaclty the same apache/php(4.3.10) install on a redhat 7.3 valhalla and apache has yet to produce the same error. Can't reproduce the problem as it seems to only occure randomly (but usually when the server is under load). Here's some server info: [root@portal6 root]# uname -a Linux portal6 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:50:31 EST 2004 i686 athlon i386 GNU/Linux [root@portal6 root]# cat /etc/issue.net Red Hat Enterprise Linux ES release 3 (Taroon Update 4) Kernel \r on an \m [root@portal6 root]# ldd /usr/local/apache/libexec/libhttpd.ep libhttpd.so => not found libm.so.6 => /lib/tls/libm.so.6 (0x00cbe000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00a2b000) libgdbm.so.2 => /usr/lib/libgdbm.so.2 (0x00593000) libexpat.so.0 => /usr/local/lib/libexpat.so.0 (0x004d1000) libdl.so.2 => /lib/libdl.so.2 (0x00743000) libc.so.6 => /lib/tls/libc.so.6 (0x00d34000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x0082d000
This is probably some memory corruption issue, possibly in mod_rewrite, possibly in the 1.3 code. To track it down further, you could try a tool like valgrind or ElectrictFence, or at least enable malloc checks in glibc; set: export MALLOC_CHECK_=2 /path/to/httpd and libc will dump core if it detects heap corruption. I'd suggest if possible not using the shared-code mode since it's so widely tested. But regardless, since this is not a bug in a Red Hat distributed package, this is not the place to report it. You could report it upstream at http://issues.apache.org/bugzilla/; if you can reproduce in a Red Hat-packaged version of Apache, please reopen.
unfortunately had no information from the malloc check. Also tried running without Zend enabled in php and the error has not yet occured (after 3 weeks) However we have tried using the apache2 RPM as supplied with this OS release - httpd-2.0.46-25.ent. just got a rogue process with the same symptoms: (gdb) backtrace #0 0x00f2eb41 in _int_malloc () from /lib/tls/libc.so.6 #1 0x00f2de9d in malloc () from /lib/tls/libc.so.6 #2 0x010dcbdc in _emalloc (size=1) at /usr/local/src/php- 4.3.10/Zend/zend_alloc.c:164 #3 0x010e92bb in zend_stack_init (stack=0x1163420) at /usr/local/src/php- 4.3.10/Zend/zend_stack.c:27 #4 0x010dd2b4 in zend_init_compiler_data_structures () at /usr/local/src/php- 4.3.10/Zend/zend_compile.c:73 #5 0x010dd372 in init_compiler () at /usr/local/src/php- 4.3.10/Zend/zend_compile.c:101 #6 0x010ea326 in zend_activate () at /usr/local/src/php-4.3.10/Zend/zend.c:636 #7 0x010c2c36 in php_request_startup () at /usr/local/src/php- 4.3.10/main/main.c:908 #8 0x010fd285 in php_apache_request_ctor (r=0x84b7cb0, ctx=0x8564850) at /usr/local/src/php-4.3.10/sapi/apache2handler/sapi_apache2.c:435 #9 0x010fd775 in php_handler (r=0x84b7cb0) at /usr/local/src/php- 4.3.10/sapi/apache2handler/sapi_apache2.c:514 #10 0x08068685 in ap_run_handler () #11 0x08068c9f in ap_invoke_handler () #12 0x08065326 in ap_process_request () #13 0x0806095c in _start () #14 0x084b7cb0 in ?? () #15 0x00000004 in ?? () #16 0x084b7cb0 in ?? () #17 0x0807228c in ap_run_pre_connection () #18 0x08072145 in ap_run_process_connection () #19 0x08066ba1 in ap_graceful_stop_signalled () #20 0x08066cf4 in ap_graceful_stop_signalled () #21 0x08066f99 in ap_graceful_stop_signalled () #22 0x08067630 in ap_mpm_run () #23 0x0806dacf in main () any thoughts?
Well, this one looks like a memory corruption issue in PHP 4.3.10 which you've compiled yourself: #0 0x00f2eb41 in _int_malloc () from /lib/tls/libc.so.6 #1 0x00f2de9d in malloc () from /lib/tls/libc.so.6 #2 0x010dcbdc in _emalloc (size=1) at /usr/local/src/php- 4.3.10/Zend/zend_alloc.c:164 #3 0x010e92bb in zend_stack_init (stack=0x1163420) at /usr/local/src/php- 4.3.10/Zend/zend_stack.c:27 But again: there's no point in reporting these issues to us unless you are using the PHP and httpd packages supplied by Red Hat. We can't help you track this issue down unless you do that. If you can reproduce the problem using the standard httpd and php packages, please re-open this bug.