Bug 892158 - Apache 2.2.15 on RHEL 6.3 segfaults with certain PHP content
Summary: Apache 2.2.15 on RHEL 6.3 segfaults with certain PHP content
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: php
Version: 6.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Web Stack Team
QA Contact: David Kutálek
URL:
Whiteboard:
Keywords:
: 895201 (view as bug list)
Depends On:
Blocks: 951075
TreeView+ depends on / blocked
 
Reported: 2013-01-05 15:09 UTC by Jared Smith
Modified: 2018-11-30 20:12 UTC (History)
5 users (show)

(edit)
Clone Of:
: 951075 (view as bug list)
(edit)
Last Closed: 2013-11-21 11:16:47 UTC


Attachments (Terms of Use)
Backtrace (with debug symbols) (4.57 KB, text/plain)
2013-01-08 15:48 UTC, Jared Smith
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:1615 normal SHIPPED_LIVE Moderate: php security, bug fix, and enhancement update 2013-11-20 21:38:52 UTC
PHP Bug Tracker 55339 None None None Never

Description Jared Smith 2013-01-05 15:09:04 UTC
Description of problem:
Apache 2.2.15 on RHEL 6.3 segfaults on certain PHP content -- it appears that PHP is segfaulting while performing query certain MySQL queries.

Version-Release number of selected component (if applicable):
[root@web1 ~]# rpm -q httpd php php-mysql
httpd-2.2.15-15.el6_2.1.x86_64
php-5.3.3-14.el6_3.x86_64
php-mysql-5.3.3-14.el6_3.x86_64

How reproducible:
Somewhat.  I've got it narrowed down to a particular virtual host that's causing problems.  Unfortunately, I can't find a rhyme or reason why some pages segfault and some don't.  Not all MySQL accesses segfault -- it seems to be something about the software I'm running that's triggering it.


Steps to Reproduce:
1. Configure a virtual host in Apache
2. Install Drupal Commerce in that virtual host, setup on MySQL
3. Some pages segfault, some don't.  It seems to be data-dependant, as I was able to do a completely fresh installation of Drupal Commerce that didn't trigger the segfault.
  
Actual results:
The apache error log shows: [Sat Jan 05 08:52:00 2013] [notice] child pid 20975 exit signal Segmentation fault (11)

When the Apache CoreDirectory directive is set, Apache produces a core dump in that directory

The backtrace for that core dump looks like:
#0  0x00007f682a688e67 in _zval_ptr_dtor () from /etc/httpd/modules/libphp5.so
#1  0x00007f682a6bf4ca in ?? () from /etc/httpd/modules/libphp5.so
#2  0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#3  0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#4  0x00007f682a6c0c01 in ?? () from /etc/httpd/modules/libphp5.so
#5  0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#6  0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#7  0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#8  0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#9  0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#10 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#11 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#12 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#13 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#14 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#15 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#16 0x00007f682a68b285 in zend_call_function () from /etc/httpd/modules/libphp5.so
#17 0x00007f682a5e3767 in ?? () from /etc/httpd/modules/libphp5.so
#18 0x00007f6827ee28d3 in xdebug_execute_internal () from /usr/lib64/php/modules/xdebug.so
#19 0x00007f682a6e2ee6 in ?? () from /etc/httpd/modules/libphp5.so
#20 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#21 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#22 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#23 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#24 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#25 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#26 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#27 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#28 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#29 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#30 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#31 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#32 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#33 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#34 0x00007f682a68b285 in zend_call_function () from /etc/httpd/modules/libphp5.so
#35 0x00007f682a68be20 in call_user_function_ex () from /etc/httpd/modules/libphp5.so
#36 0x00007f682a69513d in zend_error () from /etc/httpd/modules/libphp5.so
#37 0x00007f682a67f6fb in ?? () from /etc/httpd/modules/libphp5.so
#38 0x00007f682a669247 in ?? () from /etc/httpd/modules/libphp5.so
#39 0x00007f682a670f32 in compile_file () from /etc/httpd/modules/libphp5.so
#40 0x00007f6823833721 in ?? () from /usr/lib64/php/modules/phar.so
#41 0x00007f6827ee2ad1 in xdebug_compile_file () from /usr/lib64/php/modules/xdebug.so
#42 0x00007f682a6c5d38 in ?? () from /etc/httpd/modules/libphp5.so
#43 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#44 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#45 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#46 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#47 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#48 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#49 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#50 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#51 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#52 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#53 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#54 0x00007f682a6e2be6 in ?? () from /etc/httpd/modules/libphp5.so
#55 0x00007f682a6ba400 in execute () from /etc/httpd/modules/libphp5.so
#56 0x00007f6827ee24b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#57 0x00007f682a694b3d in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#58 0x00007f682a642da8 in php_execute_script () from /etc/httpd/modules/libphp5.so
#59 0x00007f682a71da85 in ?? () from /etc/httpd/modules/libphp5.so
#60 0x00007f6834f64b00 in ap_run_handler ()
#61 0x00007f6834f683be in ap_invoke_handler ()
#62 0x00007f6834f7386c in ap_internal_redirect ()
#63 0x00007f682c0797a5 in ?? () from /etc/httpd/modules/mod_rewrite.so
#64 0x00007f6834f64b00 in ap_run_handler ()
#65 0x00007f6834f683be in ap_invoke_handler ()
#66 0x00007f6834f73a30 in ap_process_request ()
#67 0x00007f6834f708f8 in ?? ()
#68 0x00007f6834f6c608 in ap_run_process_connection ()
#69 0x00007f6834f78807 in ?? ()
#70 0x00007f6834f78b1a in ?? ()
#71 0x00007f6834f7979c in ap_mpm_run ()
#72 0x00007f6834f50900 in main ()

When I have the xdebug module for PHP installed, I see that the last thing in the function trace before the crash is:

    0.0865   16924392                                 -> require_once(/var/www/html/commerce_kickstart-7.x-2.0/includes/database/mysql/query.inc) /var/www/html/commerce_kickstart-7.x-2.0/includes/database/database.inc:1745

I have other virtual hosts with PHP and Drupal talking to PostgreSQL without problem, so it definitely appears to be related to MySQL.

Expected results:

The PHP content loads correctly

Additional info:

This was reported to Red Hat Support under ticket 00771723.  That ticket was merged with ticket 00771972, but my hosting provider (who owns the RHEL subscription for my dedicated server) won't give me any details on ticket 00771972, so I'm reporting this here in hopes a solution can be found.

Comment 2 Remi Collet 2013-01-07 08:07:19 UTC
Can you please provides the full list of installed extension
  rpm -qa php\*
  php -m

Can you also confirm that the crash still occurs without the XDebug extension (which is not part of RHEL).

Comment 3 Jared Smith 2013-01-07 10:44:27 UTC
Yes, the problem still happens without the XDebug extension.  I installed it after the crashes started happening to see if it would help me debug the issue.

The list of PHP-related RPMS includes:

[jsmith@web1 ~]$ rpm -qa php\*
php-pdo-5.3.3-14.el6_3.x86_64
php-pgsql-5.3.3-14.el6_3.x86_64
php-xml-5.3.3-14.el6_3.x86_64
php-adodb-4.95-1.a.el6.noarch
php-common-5.3.3-14.el6_3.x86_64
php-cli-5.3.3-14.el6_3.x86_64
php-devel-5.3.3-14.el6_3.x86_64
php-mysql-5.3.3-14.el6_3.x86_64
php-soap-5.3.3-14.el6_3.x86_64
php-mbstring-5.3.3-14.el6_3.x86_64
php-pear-1.9.4-4.el6.noarch
php-pecl-xdebug-2.1.4-1.el6.x86_64
php-5.3.3-14.el6_3.x86_64
php-gd-5.3.3-14.el6_3.x86_64

And here is PHP's list of modules:

[jsmith@web1 ~]$ php -m
[PHP Modules]
bz2
calendar
Core
ctype
curl
date
dom
ereg
exif
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
json
libxml
mbstring
mysql
mysqli
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_pgsql
pdo_sqlite
pgsql
Phar
readline
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
sqlite3
standard
tokenizer
uploadprogress
wddx
xdebug
xml
xmlreader
xmlwriter
xsl
zip
zlib

[Zend Modules]
Xdebug

Comment 4 Joe Orton 2013-01-08 10:37:46 UTC
Hi Jared - do you have a backtrace from a core dump without Xdebug installed, and with php-debuginfo installed? 

There's a Zend garbage collector issue which could possibly be the cause of this, if we provide test packages would you be able to test them out?

Comment 5 Jared Smith 2013-01-08 15:21:28 UTC
Absolutely Joe-- I'd be more than happy to test out some packages... I just got the RHN administrator at my web host to enable the -debuginfo channel on this subscription, so I'm installing the debuginfo RPMs now.  I've also removed the Xdebug extension, so that we can get a backtrace without.

I'll update the bug in the next 30 minutes or so with an updated backtrace (with the debug symbols).

Comment 6 Jared Smith 2013-01-08 15:48:57 UTC
Created attachment 674917 [details]
Backtrace (with debug symbols)

Comment 7 Joe Orton 2013-01-08 16:08:27 UTC
Thanks.  This might well be unrelated to be the Zend GC issue.  The PHP code is triggering a Zend warning:

#11 0x00007fee1a47c13d in zend_error (type=8192, 
    format=0x7fee1a547070 "Call-time pass-by-reference has been deprecated")
    at /usr/src/debug/php-5.3.3/Zend/zend.c:1101

which is then invoking a Drupal-specified error handler, and the crash happens whilst evaluating that handler.  A simple workaround might be to flip "allow_call_time_pass_reference" to On in /etc/php.ini which should avoid triggering that warning in the first place.  Or there might be a fix to the Drupal code.

Comment 8 Jared Smith 2013-01-08 18:07:53 UTC
Turning off "allow_call_time_pass_reference" avoids triggering the warning and at least so far as kept httpd from segfaulting any more.  On the downside, I know longer when the code is using deprecated functions that need to be fixed :-/

Just so that I understand... are you saying that Drupal is doing something wrong to trigger that crash?  Is there something specific I should share with the Drupal developers to help them better address the issue?  Or is Drupal just hitting an unfortunate corner case in PHP which is causing the crash?

Comment 9 Remi Collet 2013-01-11 08:41:36 UTC
If think "Drupal just hitting an unfortunate corner case in PHP which is causing the crash", as any php code should never cause crash.

We need to get a simple reproducer for this issue (I cannot find one for now)

Which drupal version is it ? and how the error handler look like ?

Searching in PHP stream bug, I found some of interest
https://bugs.php.net/bug.php?id=52935 Call exit in user_error_handler cause stream relate core
https://bugs.php.net/bug.php?id=60738 Allow 'set_error_handler' to handle NULL

Comment 10 Jared Smith 2013-01-11 17:12:19 UTC
I think what I'm hitting is this particular PHP bug: https://bugs.php.net/bug.php?id=55339

The bug report includes a simple reproducer.

In looking at the PHP bug report, it appears that there is already a patch committed, but I don't know the PHP development model well enough to tell if it was simply committed to "trunk" or "head" or backported to PHP 5.3.

Comment 11 Remi Collet 2013-01-14 07:37:03 UTC
Thanks for the upstream link.

We got a simple reproducer now (bug55339.phpt) which raises a segfault:

#0  zend_switch_free (execute_data=0x7fffe7f0b268) at /usr/src/debug/php-5.3.3/Zend/zend_execute.c:382
#1  ZEND_SWITCH_FREE_SPEC_VAR_HANDLER (execute_data=0x7fffe7f0b268) at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:8392
#2  0x00000000005cc9c0 in execute (op_array=0xec05b8) at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:107
#3  0x00007ffff1e904b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#4  0x00000000005d374b in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER (execute_data=0x7fffe7f0b168)
    at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:1960
#5  0x00000000005cc9c0 in execute (op_array=0xee68f0) at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:107
#6  0x00007ffff1e904b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#7  0x000000000059d845 in zend_call_function (fci=0x7fffffff9850, fci_cache=<value optimized out>)
    at /usr/src/debug/php-5.3.3/Zend/zend_execute_API.c:963
#8  0x000000000059e3e0 in call_user_function_ex (function_table=<value optimized out>, object_pp=<value optimized out>, 
    function_name=<value optimized out>, retval_ptr_ptr=<value optimized out>, param_count=<value optimized out>, 
    params=<value optimized out>, no_separation=1, symbol_table=0x0) at /usr/src/debug/php-5.3.3/Zend/zend_execute_API.c:754
#9  0x00000000005a76fd in zend_error (type=8192, format=0x6732f0 "Call-time pass-by-reference has been deprecated")
    at /usr/src/debug/php-5.3.3/Zend/zend.c:1101
#10 0x0000000000591cbb in zend_do_pass_param (param=0x7fffffff9f60, op=67 'C', offset=2)
    at /usr/src/debug/php-5.3.3/Zend/zend_compile.c:2092
#11 0x000000000057b7a0 in zendparse () at /usr/src/debug/php-5.3.3/Zend/zend_language_parser.c:4029
#12 0x0000000000583340 in compile_string (source_string=0x7fffffffb8f0, filename=0xec1800 "") at Zend/zend_language_scanner.l:513
#13 0x00000000005d34ee in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER (execute_data=0x7fffe7f0b050)
    at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:1930
#14 0x00000000005cc9c0 in execute (op_array=0xebd7d8) at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:107
#15 0x00007ffff1e904b8 in xdebug_execute () from /usr/lib64/php/modules/xdebug.so
#16 0x00000000005a70fd in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/debug/php-5.3.3/Zend/zend.c:1194
#17 0x00000000005551d8 in php_execute_script (primary_file=0x7fffffffe260) at /usr/src/debug/php-5.3.3/main/main.c:2261
#18 0x0000000000630e35 in main (argc=2, argv=0x7fffffffe468) at /usr/src/debug/php-5.3.3/sapi/cli/php_cli.c:1192

Comment 12 Remi Collet 2013-01-14 09:57:44 UTC
I have run a test build, from RHEL-6.3 package with a single patch for this bug applied:
http://people.redhat.com/rcollet/.bug892158/

Jared can you please test it to confirm it fixes your issue.

Comment 13 Jared Smith 2013-01-14 17:11:01 UTC
I'll give those RPMs a shot later today, and let you know how they work.

Comment 14 Jared Smith 2013-01-15 01:20:22 UTC
OK, I've tested your updated RPMs, and they seem to fix the problem.  I've tried it on both the simple reproducer from the PHP bug, and on my production website.  

(As a side note, I'm having another problem with the RPMs in combination with php-pecl-apc, but I'm not sure that's being caused by these new RPMs -- let me do some more testing.)

Comment 15 Joe Orton 2013-01-16 08:34:05 UTC
*** Bug 895201 has been marked as a duplicate of this bug. ***

Comment 16 RHEL Product and Program Management 2013-01-20 06:47:21 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 18 Jared Smith 2013-01-30 18:49:44 UTC
The updated RPMs definitely fixed the problem -- any idea when this fix can be pushed into official RHEL packages?  (I've found several people seeing this problem in Fedora as well, for what it's worth, so I'd love to see this pushed to Fedora packages as well.)

Comment 19 Remi Collet 2013-01-31 06:32:35 UTC
@Jared, thanks for your feedback.

About Fedora, I'm quite surprised by your request, as the patch is a backport from upstream, it is already in Fedora packages (you can open a fedora specific bug).

Comment 20 Jared Smith 2013-02-12 18:54:36 UTC
Any status update on when the bug fixes might be available in updated RPMs for RHEL?

Comment 21 Jared Smith 2013-02-24 06:10:04 UTC
Can I please get a status update, or at a minimum, some updated RPMs with the patch applied?  I see that the RPMs in RHEL 6.4 have moved on to version php-5.3.3-22 and have several security fixes, but don't appear to have the fix identified in this bug report.  

I'm sort of stuck between using the updated RPMs and being vulnerable to the segfaults, or using older RPMs and being vulnerable to the other security issues.

Comment 22 Remi Collet 2013-02-27 11:16:50 UTC
(In reply to comment #21)
> Can I please get a status update, or at a minimum, some updated RPMs with
> the patch applied?  I see that the RPMs in RHEL 6.4 have moved on to version
> php-5.3.3-22 and have several security fixes, but don't appear to have the
> fix identified in this bug report.  

As explain in comment #16, this request was not resolved in time for the 6.4.

> I'm sort of stuck between using the updated RPMs and being vulnerable to the
> segfaults, or using older RPMs and being vulnerable to the other security
> issues.

A new test build, rebased from 6.4 packages, is available:
http://people.redhat.com/rcollet/.bug892158/

Comment 23 Jared Smith 2013-03-04 08:20:39 UTC
(In reply to comment #22)
> A new test build, rebased from 6.4 packages, is available:
> http://people.redhat.com/rcollet/.bug892158/

I finally got around to testing those pacakges this weekend -- but now I'm getting a different segfault :-(

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  zend_mm_remove_from_free_list (heap=<value optimized out>, 
    mm_block=0x7f114d3ae828)
    at /usr/src/debug/php-5.3.3/Zend/zend_alloc.c:826
826				ZEND_MM_CHECK_TREE(mm_block);


Core file is at http://web2.jaredsmith.net/core.2165, and the backtrace is at http://web2.jaredsmith.net/core.2165.backtrace.txt

Comment 24 Remi Collet 2013-03-07 08:16:25 UTC
@Jared: as this segfault seems closed to bug #910466, can you give a try to 
http://people.redhat.com/rcollet/.bug910466/

This 0.2 build includes the same fix than 0.1 and another for upstream bug #54268 (a double free during request shutdown).

Comment 26 Remi Collet 2013-04-11 12:05:53 UTC
@jared : have you see my comment 24 about 0.2 build ?

As, per your comment 14 and 18 the initial issue seems to be fixed, I propose to open a new bug the the new segfault (if not fixed by bug #910466 fix)

Comment 27 Jared Smith 2013-04-11 21:20:55 UTC
I don't have access to the 910446 bug (apparently it's marked as private), but I can't reproduce the new segfault in comment 23 very easily, so I'm not worried about it any more.  The 0.2 build in comment 24 seems to be working OK for me now, except for the PHP issue I'm still fighting a in bug 915138.

This bug can be closed.

Comment 35 errata-xmlrpc 2013-11-21 11:16:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1615.html


Note You need to log in before you can comment on or make changes to this bug.