Bug 471316
| Summary: | Apache getting segfaulting in PHP. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 4 | Reporter: | Garth <garth> |
| Component: | php | Assignee: | Joe Orton <jorton> |
| Status: | CLOSED WONTFIX | QA Contact: | BaseOS QE <qe-baseos-auto> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.7 | CC: | garth |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | i386 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-06-20 16:02:00 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Can you reproduce the problem without this third-party module loaded? #4 0xb70dd05b in _su3jdmx () from /usr/lib/php4/php_ioncube_loader_lin_4.3.so I've disabled the ioncube loader and will keep an eye on it. Still have core dumps enabled and will attach new gdb output if the crash re-occurs. Is there any additional information you would like if/when the crash re-occurs? Disabled ioncube loader. Still seeing the crash, looks to be null pointer
dereference this time (took less than 24hrs of usage to occur this time).
(gdb) bt
#0 0xb7c780e2 in _int_free () from /lib/tls/libc.so.6
#1 0xb7c7a074 in _int_realloc () from /lib/tls/libc.so.6
#2 0xb7c7b17c in realloc () from /lib/tls/libc.so.6
#3 0xb75e4291 in _erealloc (ptr=0xbfb73dd4, size=128, allow_failure=1) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_alloc.c:342
#4 0xb75f9abc in zend_hash_do_resize (ht=0xbfb6cd84) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:458
#5 0xb75fa55e in zend_hash_add_or_update (ht=0xbfc88000, arKey=0xbfb73b8c "_idmap", nKeyLength=7, pData=0xbfb73b78, nDataSize=4, pDest=0xbfc94668,
flag=1) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:299
#6 0xb75fb144 in zend_hash_copy (target=0xbfb6cd84, source=0xbfb737a4, pCopyConstructor=0xb75f44f0 <zval_add_ref>, tmp=0xbfc946a4, size=4)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:804
#7 0xb75f45b9 in _zval_copy_ctor (zvalue=0xbfb7398c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_variables.c:140
#8 0xb7601d63 in zend_assign_to_variable (result=0xb5aca984, op1=Variable "op1" is not available.
) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:501
#9 0xb76070a3 in execute (op_array=0xb5ac6d0c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1383
#10 0xb76076b2 in execute (op_array=0xbfa3865c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#11 0xb76076b2 in execute (op_array=0xbfa385f4) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#12 0xb76076b2 in execute (op_array=0xbf8b3914) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#13 0xb76076b2 in execute (op_array=0xbf8b38ac) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#14 0xb76076b2 in execute (op_array=0xbf616264) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#15 0xb75f5db0 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend.c:900
#16 0xb75c696f in php_execute_script (primary_file=0xbfca29b0) at /usr/src/redhat/BUILD/php-4.3.9/main/main.c:1753
#17 0xb7611ce1 in php_handler (r=0xbf5dd090) at /usr/src/redhat/BUILD/php-4.3.9/sapi/apache2handler/sapi_apache2.c:575
#18 0xb7f7c0c8 in ap_run_handler (r=0xbf5dd090) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:156
#19 0xb7f7c5ec in ap_invoke_handler (r=0xbf5dd090) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:368
#20 0xb7f78d2e in ap_process_request (r=0xbf5dd090) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_request.c:246
#21 0xb7f736d8 in ap_process_http_connection (c=0xbf5d6e20) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_core.c:250
#22 0xb7f87148 in ap_run_process_connection (c=0xbf5d6e20) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:42
#23 0xb7f8751f in ap_process_connection (c=0xbf5d6e20, csd=0xbf5d6d48) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:175
#24 0xb7f79e29 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:609
---Type <return> to continue, or q <return> to quit---
#25 0xb7f7a07c in make_child (s=0xb7fb0e78, slot=16) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:703
#26 0xb7f7a835 in ap_mpm_run (_pconf=0xb7faf0a8, plog=0xb7fdb158, s=0xb7fb0e78) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:838
#27 0xb7f81dd9 in main (argc=1, argv=0xbfca2f54) at /usr/src/redhat/BUILD/httpd-2.0.52/server/main.c:618
(gdb) x/i $eip
0xb7c780e2 <_int_free+226>: cmp %ecx,0xc(%edx)
(gdb) info reg edx
edx 0x0 0
(gdb) frame 16
#16 0xb75c696f in php_execute_script (primary_file=0xbfca29b0) at /usr/src/redhat/BUILD/php-4.3.9/main/main.c:1753
1753 /usr/src/redhat/BUILD/php-4.3.9/main/main.c: No such file or directory.
in /usr/src/redhat/BUILD/php-4.3.9/main/main.c
(gdb) x/10x 0xbfca29b0
0xbfca29b0: 0x00000001 0xbf5dea90 0xbf6162cc 0x000004c3
0xbfca29c0: 0x00000000 0x00000010 0x00000000 0xb7836b50
0xbfca29d0: 0xbf5f4228 0xbf5f4098
(gdb) x/s 0xbf5dea90
0xbf5dea90: "/usr/share/psa-horde/imp/message.php"
(gdb)
NOTE: Simply accessing this horde message.php script will not replicate the crash.
That null ptr deref, was the second crash with ioncube disabled. Missed this one
when I was looking at the timestamps on the core files.
Looks to be same issue as the original report,
double free() or corrupted malloc chunk header:
(gdb) bt
#0 0xb7c78935 in free () from /lib/tls/libc.so.6
#1 0xb75e3ffa in _efree (ptr=0xbfc8cd64) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_alloc.c:271
#2 0xb75f44cb in _zval_dtor (zvalue=0xbfc819f8) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_variables.c:61
#3 0xb75eda8e in destroy_op_array (op_array=0xbfc79cec) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:165
#4 0xb75edb08 in destroy_zend_function (function=0x0) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:100
#5 0xb75faae8 in zend_hash_destroy (ht=0xb5c65bf8) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:563
#6 0xb75ed976 in destroy_zend_class (ce=0xb5c65be0) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:124
#7 0xb75fa9c2 in zend_hash_del_key_or_index (ht=0xbbeb6d38, arKey=0xbfc8b548 "wpdb", nKeyLength=5, h=190435940, flag=0)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:534
#8 0xb75fb074 in zend_hash_reverse_apply (ht=0xbbeb6d38, apply_func=0xb75eb100 <is_not_internal_class>)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_hash.c:782
#9 0xb75eb68b in shutdown_executor () at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute_API.c:202
#10 0xb75f566b in zend_deactivate () at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend.c:667
#11 0xb75c4a49 in php_request_shutdown (dummy=0x0) at /usr/src/redhat/BUILD/php-4.3.9/main/main.c:1008
#12 0xb7611bfe in php_handler (r=0xbf5415b0) at /usr/src/redhat/BUILD/php-4.3.9/sapi/apache2handler/sapi_apache2.c:461
#13 0xb7f7c0c8 in ap_run_handler (r=0xbf5415b0) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:156
#14 0xb7f7c5ec in ap_invoke_handler (r=0xbf5415b0) at /usr/src/redhat/BUILD/httpd-2.0.52/server/config.c:368
#15 0xb7f78d2e in ap_process_request (r=0xbf5415b0) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_request.c:246
#16 0xb7f736d8 in ap_process_http_connection (c=0xbf53b340) at /usr/src/redhat/BUILD/httpd-2.0.52/modules/http/http_core.c:250
#17 0xb7f87148 in ap_run_process_connection (c=0xbf53b340) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:42
#18 0xb7f8751f in ap_process_connection (c=0xbf53b340, csd=0xbf53b268) at /usr/src/redhat/BUILD/httpd-2.0.52/server/connection.c:175
#19 0xb7f79e29 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:609
#20 0xb7f7a07c in make_child (s=0xb7fb0e78, slot=7) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:703
#21 0xb7f7a835 in ap_mpm_run (_pconf=0xb7faf0a8, plog=0xb7fdb158, s=0xb7fb0e78) at /usr/src/redhat/BUILD/httpd-2.0.52/server/mpm/prefork/prefork.c:838
#22 0xb7f81dd9 in main (argc=1, argv=0xbfca2f54) at /usr/src/redhat/BUILD/httpd-2.0.52/server/main.c:618
(gdb) x/i $eip
0xb7c78935 <free+117>: cmpxchg %ecx,(%esi)
(gdb) info reg esi
esi 0x7274 29300
After this occurs, apache seems to be very unstable and starts crashing all
over the place, until restarted, at which point it is fine again.
So you can probably ignore that null-ptr deref post above, probably a red herring.
Please let me know if you want any more information.
Also, note: I can't paste the details here (for privacy reasons, it's a clients site). But going to frame #15 and getting the incoming request url and method then re-running it did not replicate the crash. Ran it several times and no crash. Can you narrow this down to a specific script which fails? It's going to be hard for us to fix this without being able to replicate it internally. On a development server (not in production!) it can help to run httpd in single-process mode with malloc debugging enabled, as follows, to replicate such crashes: # service httpd stop # export MALLOC_CHECK_=2 # gdb /usr/sbin/httpd -X ... (run) Yeah. Unfortunately, I've so far been unable to replicate it. The crash is now no longer happening after the kernel was updated. I will update the ticket if I can replicate it on a dev box. Did you manage to replicate this in the end? Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. Please See https://access.redhat.com/support/policy/updates/errata/ If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue. |
Description of problem: Apache is crashing, bug seems to be in php. Problem occurs after about 12-48hrs of uptime. Looks like double _efree(). (gdb) bt #0 0xb7565fb7 in _efree (ptr=0xbfa06fc4) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_alloc.c:265 #1 0xb75764cb in _zval_dtor (zvalue=0xb50a3bc4) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_variables.c:61 #2 0xb756fa9d in destroy_op_array (op_array=0xb5a7686c) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_opcode.c:159 #3 0xb7589f02 in execute (op_array=0xb5a963fc) at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:2231 #4 0xb70dd05b in _su3jdmx () from /usr/lib/php4/php_ioncube_loader_lin_4.3.so #5 0xb5a963fc in ?? () #6 0x00000000 in ?? () (gdb) x/i $eip 0xb7589f02 <execute+13986>: mov 0xfffffdb0(%ebp),%edx (gdb) info reg edx edx 0xb7cb7bf1 -1211401231 (gdb) x/s 0xb7cb7bf1 0xb7cb7bf1 <__libc_ptyname1+5130>: "corrupted double-linked list" Bug maybe fixed already upstream. Version-Release number of selected component (if applicable): php-4.3.9-3.22.12 PHP 4.3.9 (cgi) (built: Sep 9 2008 11:12:20) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies with the ionCube PHP Loader v3.1.16, Copyright (c) 2002-2006, by ionCube Ltd. httpd-2.0.52-41.ent Server version: Apache/2.0.52 Server built: Sep 9 2008 11:00:12 Server's Module Magic Number: 20020903:9 Architecture: 32-bit Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" How reproducible: Have not been able to trigger the bug on demand, but it will occur with 48hrs of apache restart. Busy production shared hosting webserver (2000+ sites). Steps to Reproduce: 1. 2. 3. Actual results: Apache children exit with Signal 11. Expected results: Apache should not segfault. Additional info: