Bug 471316 - Apache getting segfaulting in PHP.
Apache getting segfaulting in PHP.
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: php (Show other bugs)
4.7
i386 Linux
medium Severity high
: rc
: ---
Assigned To: Joe Orton
BaseOS QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-12 17:33 EST by Garth
Modified: 2012-06-20 12:02 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-20 12:02:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Garth 2008-11-12 17:33:08 EST
Description of problem:
Apache is crashing, bug seems to be in php. 
Problem occurs after about 12-48hrs of uptime.

Looks like double _efree().

(gdb) bt
#0  0xb7565fb7 in _efree (ptr=0xbfa06fc4) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_alloc.c:265
#1  0xb75764cb in _zval_dtor (zvalue=0xb50a3bc4) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_variables.c:61
#2  0xb756fa9d in destroy_op_array (op_array=0xb5a7686c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:159
#3  0xb7589f02 in execute (op_array=0xb5a963fc) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:2231
#4  0xb70dd05b in _su3jdmx () from /usr/lib/php4/php_ioncube_loader_lin_4.3.so
#5  0xb5a963fc in ?? ()
#6  0x00000000 in ?? ()
(gdb) x/i $eip
0xb7589f02 <execute+13986>:	mov    0xfffffdb0(%ebp),%edx
(gdb) info reg edx
edx            0xb7cb7bf1	-1211401231
(gdb) x/s 0xb7cb7bf1
0xb7cb7bf1 <__libc_ptyname1+5130>:	 "corrupted double-linked list"

Bug maybe fixed already upstream.


Version-Release number of selected component (if applicable):
php-4.3.9-3.22.12

PHP 4.3.9 (cgi) (built: Sep  9 2008 11:12:20)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
    with the ionCube PHP Loader v3.1.16, Copyright (c) 2002-2006, by ionCube Ltd.

httpd-2.0.52-41.ent

Server version: Apache/2.0.52
Server built:   Sep  9 2008 11:00:12
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"



How reproducible:
Have not been able to trigger the bug on demand, but it will occur with 48hrs
of apache restart. Busy production shared hosting webserver (2000+ sites).


Steps to Reproduce:
1.
2.
3.
  
Actual results:
Apache children exit with Signal 11. 

Expected results:
Apache should not segfault.

Additional info:
Comment 1 Joe Orton 2008-11-16 16:22:32 EST
Can you reproduce the problem without this third-party module loaded?

#4  0xb70dd05b in _su3jdmx () from /usr/lib/php4/php_ioncube_loader_lin_4.3.so
Comment 2 Garth 2008-11-17 17:16:32 EST
I've disabled the ioncube loader and will keep an eye on it.
Still have core dumps enabled and will attach new gdb output if the crash
re-occurs.

Is there any additional information you would like if/when the crash 
re-occurs?
Comment 3 Garth 2008-11-18 17:34:07 EST
Disabled ioncube loader. Still seeing the crash, looks to be null pointer
dereference this time (took less than 24hrs of usage to occur this time).


(gdb) bt
#0  0xb7c780e2 in _int_free () from /lib/tls/libc.so.6
#1  0xb7c7a074 in _int_realloc () from /lib/tls/libc.so.6
#2  0xb7c7b17c in realloc () from /lib/tls/libc.so.6
#3  0xb75e4291 in _erealloc (ptr=0xbfb73dd4, size=128, allow_failure=1) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_alloc.c:342
#4  0xb75f9abc in zend_hash_do_resize (ht=0xbfb6cd84) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:458
#5  0xb75fa55e in zend_hash_add_or_update (ht=0xbfc88000, arKey=0xbfb73b8c "_idmap", nKeyLength=7, pData=0xbfb73b78, nDataSize=4, pDest=0xbfc94668, 
    flag=1) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:299
#6  0xb75fb144 in zend_hash_copy (target=0xbfb6cd84, source=0xbfb737a4, pCopyConstructor=0xb75f44f0 <zval_add_ref>, tmp=0xbfc946a4, size=4)
    at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:804
#7  0xb75f45b9 in _zval_copy_ctor (zvalue=0xbfb7398c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_variables.c:140
#8  0xb7601d63 in zend_assign_to_variable (result=0xb5aca984, op1=Variable "op1" is not available.
) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:501
#9  0xb76070a3 in execute (op_array=0xb5ac6d0c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1383
#10 0xb76076b2 in execute (op_array=0xbfa3865c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#11 0xb76076b2 in execute (op_array=0xbfa385f4) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#12 0xb76076b2 in execute (op_array=0xbf8b3914) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#13 0xb76076b2 in execute (op_array=0xbf8b38ac) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#14 0xb76076b2 in execute (op_array=0xbf616264) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#15 0xb75f5db0 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend.c:900
#16 0xb75c696f in php_execute_script (primary_file=0xbfca29b0) at /usr/src/redhat/BUILD/php-4.3.9/main/main.c:1753
#17 0xb7611ce1 in php_handler (r=0xbf5dd090) at /usr/src/redhat/BUILD/php-4.3.9/sapi/apache2handler/sapi_apache2.c:575
#18 0xb7f7c0c8 in ap_run_handler (r=0xbf5dd090) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:156
#19 0xb7f7c5ec in ap_invoke_handler (r=0xbf5dd090) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:368
#20 0xb7f78d2e in ap_process_request (r=0xbf5dd090) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_request.c:246
#21 0xb7f736d8 in ap_process_http_connection (c=0xbf5d6e20) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_core.c:250
#22 0xb7f87148 in ap_run_process_connection (c=0xbf5d6e20) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:42
#23 0xb7f8751f in ap_process_connection (c=0xbf5d6e20, csd=0xbf5d6d48) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:175
#24 0xb7f79e29 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:609
---Type <return> to continue, or q <return> to quit---
#25 0xb7f7a07c in make_child (s=0xb7fb0e78, slot=16) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:703
#26 0xb7f7a835 in ap_mpm_run (_pconf=0xb7faf0a8, plog=0xb7fdb158, s=0xb7fb0e78) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:838
#27 0xb7f81dd9 in main (argc=1, argv=0xbfca2f54) at /usr/src/redhat/BUILD/httpd-2.0.52/server/main.c:618
(gdb) x/i $eip
0xb7c780e2 <_int_free+226>:	cmp    %ecx,0xc(%edx)
(gdb) info reg edx
edx            0x0	0
(gdb) frame 16
#16 0xb75c696f in php_execute_script (primary_file=0xbfca29b0) at /usr/src/redhat/BUILD/php-4.3.9/main/main.c:1753
1753	/usr/src/redhat/BUILD/php-4.3.9/main/main.c: No such file or directory.
	in /usr/src/redhat/BUILD/php-4.3.9/main/main.c
(gdb) x/10x 0xbfca29b0
0xbfca29b0:	0x00000001	0xbf5dea90	0xbf6162cc	0x000004c3
0xbfca29c0:	0x00000000	0x00000010	0x00000000	0xb7836b50
0xbfca29d0:	0xbf5f4228	0xbf5f4098
(gdb) x/s 0xbf5dea90
0xbf5dea90:	 "/usr/share/psa-horde/imp/message.php"
(gdb) 
 
NOTE: Simply accessing this horde message.php script will not replicate the crash.
Comment 4 Garth 2008-11-18 17:56:20 EST
That null ptr deref, was the second crash with ioncube disabled. Missed this one
when I was looking at the timestamps on the core files.
Looks to be same issue as the original report,
double free() or corrupted malloc chunk header:

(gdb) bt
#0  0xb7c78935 in free () from /lib/tls/libc.so.6
#1  0xb75e3ffa in _efree (ptr=0xbfc8cd64) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_alloc.c:271
#2  0xb75f44cb in _zval_dtor (zvalue=0xbfc819f8) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_variables.c:61
#3  0xb75eda8e in destroy_op_array (op_array=0xbfc79cec) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:165
#4  0xb75edb08 in destroy_zend_function (function=0x0) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:100
#5  0xb75faae8 in zend_hash_destroy (ht=0xb5c65bf8) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:563
#6  0xb75ed976 in destroy_zend_class (ce=0xb5c65be0) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:124
#7  0xb75fa9c2 in zend_hash_del_key_or_index (ht=0xbbeb6d38, arKey=0xbfc8b548 "wpdb", nKeyLength=5, h=190435940, flag=0)
    at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:534
#8  0xb75fb074 in zend_hash_reverse_apply (ht=0xbbeb6d38, apply_func=0xb75eb100 <is_not_internal_class>)
    at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:782
#9  0xb75eb68b in shutdown_executor () at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute_API.c:202
#10 0xb75f566b in zend_deactivate () at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend.c:667
#11 0xb75c4a49 in php_request_shutdown (dummy=0x0) at /usr/src/redhat/BUILD/php-4.3.9/main/main.c:1008
#12 0xb7611bfe in php_handler (r=0xbf5415b0) at /usr/src/redhat/BUILD/php-4.3.9/sapi/apache2handler/sapi_apache2.c:461
#13 0xb7f7c0c8 in ap_run_handler (r=0xbf5415b0) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:156
#14 0xb7f7c5ec in ap_invoke_handler (r=0xbf5415b0) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:368
#15 0xb7f78d2e in ap_process_request (r=0xbf5415b0) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_request.c:246
#16 0xb7f736d8 in ap_process_http_connection (c=0xbf53b340) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_core.c:250
#17 0xb7f87148 in ap_run_process_connection (c=0xbf53b340) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:42
#18 0xb7f8751f in ap_process_connection (c=0xbf53b340, csd=0xbf53b268) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:175
#19 0xb7f79e29 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:609
#20 0xb7f7a07c in make_child (s=0xb7fb0e78, slot=7) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:703
#21 0xb7f7a835 in ap_mpm_run (_pconf=0xb7faf0a8, plog=0xb7fdb158, s=0xb7fb0e78) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:838
#22 0xb7f81dd9 in main (argc=1, argv=0xbfca2f54) at /usr/src/redhat/BUILD/httpd-2.0.52/server/main.c:618
(gdb) x/i $eip
0xb7c78935 <free+117>:	cmpxchg %ecx,(%esi)
(gdb) info reg esi
esi            0x7274	29300


After this occurs, apache seems to be very unstable and starts crashing all
over the place, until restarted, at which point it is fine again.
So you can probably ignore that null-ptr deref post above, probably a red herring.

Please let me know if you want any more information.
Comment 5 Garth 2008-11-18 19:54:28 EST
Also, note: I can't paste the details here (for privacy reasons, it's a clients
site). But going to frame #15 and getting the incoming request url and method
then re-running it did not replicate the crash. Ran it several times and
no crash.
Comment 6 Joe Orton 2008-12-18 10:28:11 EST
Can you narrow this down to a specific script which fails?

It's going to be hard for us to fix this without being able to replicate it internally.

On a development server (not in production!) it can help to run httpd in single-process mode with malloc debugging enabled, as follows, to replicate such crashes:

# service httpd stop
# export MALLOC_CHECK_=2
# gdb /usr/sbin/httpd -X
...
(run)
Comment 7 Garth 2008-12-21 18:06:16 EST
Yeah. Unfortunately, I've so far been unable to replicate it.
The crash is now no longer happening after the kernel was updated.
I will update the ticket if I can replicate it on a dev box.
Comment 8 Joe Orton 2009-01-26 05:58:20 EST
Did you manage to replicate this in the end?
Comment 9 Jiri Pallich 2012-06-20 12:02:00 EDT
Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. 
Please See https://access.redhat.com/support/policy/updates/errata/

If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.

Note You need to log in before you can comment on or make changes to this bug.