Description of problem: Customer is seeing segmenation faults in httpd.worker in configuration where primary work is mod_proxy to backend servers. Version-Release number of selected component (if applicable): httpd-2.2.3-31.el5-i386 apr-1.2.7-11.el5_3.1-i386 apr-util-1.2.7-7.el5_3.2-i386 Appears to correspond with upstream bug, "Worker segmentation fault" https://issues.apache.org/bugzilla/show_bug.cgi?id=45792
In core.15562, thread 65 #0 0x0054e410 in __kernel_vsyscall () #1 0x00342226 in kill () from /lib/libc.so.6 #2 0x0089e76e in sig_coredump (sig=11) at /usr/src/debug/httpd-2.2.3/server/mpm_common.c:1204 #3 <signal handler called> #4 0x786f7270 in ?? () #5 0x008939f2 in ap_core_output_filter (f=0x90d6c50, b=0x90da648) at /usr/src/debug/httpd-2.2.3/server/core_filters.c:630 #6 0x008a04a0 in ap_pass_brigade (next=0x1, bb=0x9148578) at /usr/src/debug/httpd-2.2.3/server/util_filter.c:526 #7 0x008a47eb in ap_process_request (r=0x90e2648) at /usr/src/debug/httpd-2.2.3/modules/http/http_request.c:231 #8 0x008a163f in ap_process_http_connection (c=0x90d67d0) at /usr/src/debug/httpd-2.2.3/modules/http/http_core.c:184 #9 0x0089d01d in ap_run_process_connection (c=0x90d67d0) at /usr/src/debug/httpd-2.2.3/server/connection.c:43 #10 0x0089d11c in ap_process_connection (c=0x90d67d0, csd=0x90d6620) at /usr/src/debug/httpd-2.2.3/server/connection.c:178 #11 0x008a9eff in worker_thread (thd=0x9014c80, dummy=0x8fbdea8) at /usr/src/debug/httpd-2.2.3/server/mpm/worker/worker.c:544 #12 0x009a6c46 in dummy_worker (opaque=0x9014c80) at threadproc/unix/thread.c:138 #13 0x00b6c5ab in start_thread (arg=<value optimized out>) at pthread_create.c:297 #14 0x003eacfe in clone () from /lib/libc.so.6 from that, in frame 5 (gdb) frame 5 #5 0x008939f2 in ap_core_output_filter (f=0x90d6c50, b=0x90da648) at /usr/src/debug/httpd-2.2.3/server/core_filters.c:630 630 rv = apr_bucket_read(e, &str, &n, eblock); (gdb) dump_brigade b dump of brigade 0x90da648 | type (address) | length | data addr | contents | rc -------------------------------------------------------------------------------- 0 | HEAP (0x090d9580) | 274 | 0x090d9528 | [HTTP/1.1 500 Inte...] | 1 1 | Cannot access memory at address 0x7948a0 We have a corrupted bucket brigade chain.
For core.21501 the failure is again in thread 65 (gdb) thread 65 [Switching to thread 65 (process 21504)]#0 0x0054e410 in __kernel_vsyscall () (gdb) bt #0 0x0054e410 in __kernel_vsyscall () #1 0x00342226 in kill () from /lib/libc.so.6 #2 0x0089e76e in sig_coredump (sig=11) at /usr/src/debug/httpd-2.2.3/server/mpm_common.c:1204 #3 <signal handler called> #4 0x0917cf40 in ?? () #5 0x00a5754f in apr_brigade_cleanup (data=0x90f2a78) at buckets/apr_brigade.c:44 #6 0x00a57c3a in apr_brigade_destroy (b=0x90f2a78) at buckets/apr_brigade.c:53 #7 0x00893fcb in ap_core_output_filter (f=0x90eb050, b=0x90f2a78) at /usr/src/debug/httpd-2.2.3/server/core_filters.c:882 #8 0x008a04a0 in ap_pass_brigade (next=0x917cf58, bb=0x90f85a0) at /usr/src/debug/httpd-2.2.3/server/util_filter.c:526 #9 0x008a47eb in ap_process_request (r=0x90eea28) at /usr/src/debug/httpd-2.2.3/modules/http/http_request.c:231 #10 0x008a163f in ap_process_http_connection (c=0x90eabd0) at /usr/src/debug/httpd-2.2.3/modules/http/http_core.c:184 #11 0x0089d01d in ap_run_process_connection (c=0x90eabd0) at /usr/src/debug/httpd-2.2.3/server/connection.c:43 #12 0x0089d11c in ap_process_connection (c=0x90eabd0, csd=0x90eaa20) at /usr/src/debug/httpd-2.2.3/server/connection.c:178 #13 0x008a9eff in worker_thread (thd=0x9014c80, dummy=0x8fbdea8) at /usr/src/debug/httpd-2.2.3/server/mpm/worker/worker.c:544 #14 0x009a6c46 in dummy_worker (opaque=0x9014c80) at threadproc/unix/thread.c:138 #15 0x00b6c5ab in start_thread (arg=<value optimized out>) at pthread_create.c:297 #16 0x003eacfe in clone () from /lib/libc.so.6 In this we're trying to clean up the bucket brigade which again is corrupt (gdb) frame 7 #7 0x00893fcb in ap_core_output_filter (f=0x90eb050, b=0x90f2a78) at /usr/src/debug/httpd-2.2.3/server/core_filters.c:882 882 apr_brigade_destroy(b); (gdb) dump_brigade b dump of brigade 0x90f2a78 | type (address) | length | data addr | contents | rc -------------------------------------------------------------------------------- 0 | HEAP (0x0917d080) | 0 | 0x0917d0d8 | [] | 1 1 | SOCKET (0x0917d130) | -1 | 0x0917af48 | [**unprintable**] 2 | È (0x0917b5ac) | 152547088 | 0x0917cf58 | [**unknown**] | n/a 3 | HEAP (0x0917d080) | 0 | 0x0917d0d8 | [] | 1 4 | SOCKET (0x0917d130) | -1 | 0x0917af48 | [**unprintable**] 5 | È (0x0917b5ac) | 152547088 | 0x0917cf58 | [**unknown**] | n/a and a trace which loops.
core.30231 where the action is in thread 60 (gdb) thread 60 [Switching to thread 60 (process 30239)]#0 0x0054e410 in __kernel_vsyscall () (gdb) bt #0 0x0054e410 in __kernel_vsyscall () #1 0x00342226 in kill () from /lib/libc.so.6 #2 0x0089e76e in sig_coredump (sig=11) at /usr/src/debug/httpd-2.2.3/server/mpm_common.c:1204 #3 <signal handler called> #4 0x008939ef in ap_core_output_filter (f=0x9171620, b=0x92697b8) at /usr/src/debug/httpd-2.2.3/server/core_filters.c:630 #5 0x008a04a0 in ap_pass_brigade (next=0x1, bb=0x9180618) at /usr/src/debug/httpd-2.2.3/server/util_filter.c:526 #6 0x008a47eb in ap_process_request (r=0x9179008) at /usr/src/debug/httpd-2.2.3/modules/http/http_request.c:231 #7 0x008a163f in ap_process_http_connection (c=0x91711a0) at /usr/src/debug/httpd-2.2.3/modules/http/http_core.c:184 #8 0x0089d01d in ap_run_process_connection (c=0x91711a0) at /usr/src/debug/httpd-2.2.3/server/connection.c:43 #9 0x0089d11c in ap_process_connection (c=0x91711a0, csd=0x9170ff0) at /usr/src/debug/httpd-2.2.3/server/connection.c:178 #10 0x008a9eff in worker_thread (thd=0x9014d20, dummy=0x8fc0b08) at /usr/src/debug/httpd-2.2.3/server/mpm/worker/worker.c:544 #11 0x009a6c46 in dummy_worker (opaque=0x9014d20) at threadproc/unix/thread.c:138 #12 0x00b6c5ab in start_thread (arg=<value optimized out>) at pthread_create.c:297 #13 0x003eacfe in clone () from /lib/libc.so.6 and examining the bucket brigade at the top of the set (gdb) frame 4 #4 0x008939ef in ap_core_output_filter (f=0x9171620, b=0x92697b8) at /usr/src/debug/httpd-2.2.3/server/core_filters.c:630 630 rv = apr_bucket_read(e, &str, &n, eblock); (gdb) dump_brigade b dump of brigade 0x92697b8 | type (address) | length | data addr | contents | rc -------------------------------------------------------------------------------- 0 | HEAP (0x09173ea0) | 274 | 0x09173e48 | [HTTP/1.1 500 Inte...] | 1 1 | Cannot access memory at address 0x63727320 another corrupt bucket brigade.
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
For anybody seeing this problem, a test repo is available with experimental and *unsupported* packages which contain a fix for this issue: http://people.redhat.com/jorton/Tikanga-httpd/
*** Bug 701972 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-1067.html