Bug 1289457 - httpd segfault in php_module_shutdown when opcache loaded twice
Summary: httpd segfault in php_module_shutdown when opcache loaded twice
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: php   
(Show other bugs)
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Web Stack Team
QA Contact: David Kutálek
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1203710 1313485 1289025 1295829
TreeView+ depends on / blocked
 
Reported: 2015-12-08 07:54 UTC by Hisanobu Okuda
Modified: 2016-11-03 21:07 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-03 21:07:26 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2598 normal SHIPPED_LIVE Moderate: php security and bug fix update 2016-11-03 12:12:00 UTC
PHP Bug Tracker 71089 None None None Never

Description Hisanobu Okuda 2015-12-08 07:54:53 UTC
Description of problem:
httpd died with segfault.

The backtrace is :-

(gdb) bt
#0  0x00007f808cfd15f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f808cfd2ce8 in __GI_abort () at abort.c:90
#2  0x00007f808d011317 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f808d11a9c8 "*** Error in `%s': %s: 0x%s ***\n
") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196                                                                                  
#3  0x00007f808d018fe1 in malloc_printerr (ar_ptr=0x7f808d356760 <main_arena>, ptr=<optimized out>, str=0x7f808d1180b4 "free(): inva
lid pointer", action=3) at malloc.c:5013                                                                                           
#4  _int_free (av=0x7f808d356760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3835
#5  0x00007f807ff1d47b in php_module_shutdown () at /usr/src/debug/php-5.4.16/main/main.c:2367
#6  0x00007f807ff1d539 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.16/main/main.c:2335
#7  0x00007f80800292d1 in php_apache_server_shutdown (tmp=<optimized out>) at /usr/src/debug/php-5.4.16/sapi/apache2handler/sapi_apa
che2.c:388                                                                                                                         
#8  0x00007f808d79640e in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
#9  apr_pool_clear (pool=pool@entry=0x7f808fec8138) at memory/unix/apr_pools.c:772
#10 0x00007f808eabc9ab in main (argc=2, argv=0x7ffe2d2b29c8) at main.c:707
(gdb) 

Version-Release number of selected component (if applicable):
httpd-2.4.6-40.el7.x86_64
php-5.4.16-36.el7_1.x86_64

How reproducible:
Not identified how to reproduce the issues as it just happens over time, around a day.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Joe Orton 2015-12-09 08:41:57 UTC
Without a repro case this type of php crash is hard to track down.

Comment 3 Remi Collet 2015-12-09 09:41:49 UTC
This looks like a refcount issue, as Joe said, very hard to track because the issue raise the segfault later, during shutdown.

Notice: as this occurs in shutdown, the request have been served.

A list of installed extensions may help ("rpm -qa php\* | sort" and "php -m")

Could also be related to gc.

A possible workaround is to disable gc (zend.enable_gc=0), but this may require to increase memory_limit.

Comment 4 Hisanobu Okuda 2015-12-09 13:54:40 UTC
Joe and Remi, I really appreciate your prompt response. I will try to gather up info to reproduce the issue.

Comment 5 Hisanobu Okuda 2015-12-10 10:47:01 UTC
@Joe, Remi, I could reproduce the issue and found the cause, the workaround, and how to fix.

The issue is reproduced when stopping httpd/php with *opcache*. Installing the php-pecl-zendopcache package, you can reproduce the issue simply running `systemcrl start httpd` then `systemctl stop httpd`.

The issue occurs when freeing CG(interned_strings_start) in zend_interned_strings_dtor(). The code is as follows:-

/usr/src/debug/php-5.4.16/Zend/zend_string.c:
------------------------------------------------------
void zend_interned_strings_dtor(TSRMLS_D)
{
#ifndef ZTS
#if ZEND_DEBUG_INTERNED_STRINGS
	mprotect(CG(interned_strings_start), CG(interned_strings_end) - CG(interned_strings_start), PROT_WRITE | PROT_READ);
#endif
	free(CG(interned_strings).arBuckets);
	free(CG(interned_strings_start)); <<===HERE
#endif
}
------------------------------------------------------

backtrace:
------------------------------------------------------
#0  zend_interned_strings_dtor () at /usr/src/debug/php-5.4.16/Zend/zend_string.c:78
#1  0x00007f4d8988f47b in php_module_shutdown () at /usr/src/debug/php-5.4.16/main/main.c:2367
#2  0x00007f4d8988f539 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.16/main/main.c:2335
#3  0x00007f4d8999ab21 in php_apache_child_shutdown (tmp=<optimized out>)
    at /usr/src/debug/php-5.4.16/sapi/apache2handler/sapi_apache2.c:398
#4  0x00007f4d976f61ae in apr_pool_destroy () from /lib64/libapr-1.so.0
#5  0x00007f4d8e41823c in clean_child_exit (code=code@entry=0) at prefork.c:221
#6  0x00007f4d8e4186e7 in child_main (child_num_arg=child_num_arg@entry=0) at prefork.c:728
#7  0x00007f4d8e418a55 in make_child (s=0x7f4d99227340, slot=slot@entry=0) at prefork.c:810
#8  0x00007f4d8e418ab6 in startup_children (number_to_start=1) at prefork.c:828
#9  0x00007f4d8e4197c0 in prefork_run (_pconf=<optimized out>, plog=0x7f4d9922b378, s=0x7f4d99227340) at prefork.c:986
#10 0x00007f4d98a2350e in ap_run_mpm (pconf=0x7f4d991fe158, plog=0x7f4d9922b378, s=0x7f4d99227340) at mpm_common.c:96
#11 0x00007f4d98a1cb36 in main (argc=2, argv=0x7fffb4909928) at main.c:777
------------------------------------------------------

If opcache is not installed, CG(interned_strings_start) points a memory chunk which is normally malloc-ed. Therefore, freeing it works fine. If opcache is installed, CG(interned_strings_start) is replaced with a pointer to a "zend_shared_alloc"-ed memory chunk in zend_accel_init_shm() in /usr/src/debug/php-pecl-zendopcache-7.0.5/NTS/ZendAccelerator.c. Since the chunk is not a malloc-ed one, freeing it leads to ABRT.

The workaround is to set 0 to opcache.interned_strings_buffer in /etc/php.d/opcache.ini. Since it disables the "interning strings" feature, it may affect memory usage and through-put, but it is better than disabling opcache itself.

The fix was provided in [1] which is applied into the php core (not opcache). I found the same issue was filed in [2]. I know the php-pecl-zendopcache package is provided in EPEL repository and not fully supported as of now. However, I think we should merge the fix, because the fix is small and surely harmless, testing it is very easy (just stop httpd), and it is safe for future when the opcache is fully supported, and to top it off, our customer needs opcache. Please merge it in the next build.

[1] https://bugs.php.net/patch-display.php?bug=65338&patch=zend_interned_strings_shutdown_AV&revision=1374773456
[2] https://bugs.php.net/bug.php?id=65338

Comment 6 Hisanobu Okuda 2015-12-10 10:49:35 UTC
Reproduced using:

php-5.4.16-36.el7_1.x86_64
php-pecl-zendopcache-7.0.5-1.el7.x86_64

Comment 7 Remi Collet 2015-12-10 14:40:58 UTC
@Hisanobu: great thanks for your analysis.

Strangely, the patch which is supposed to fix upstream bug 65338 is part opcache (7.0.3-dev), so is present in 7.0.5.

I will dig further on this;

Comment 8 Remi Collet 2015-12-10 15:33:41 UTC
Some testing:

Startup:
- zend_interned_string_init => alloc
- zend_accel_init_shm => save initial value + reallocation

Shutdown:
- accel_shutdown => restore initial value
- zend_interned_string_dtor => free

So from my test, cannot raise the issue (but with 5.5, will try the same with 5.4.16 asap)

@Hisanobu can you please ensure there is no other extension ? (ex: APC... which also play with interned strings...): php -m

Comment 9 Hisanobu Okuda 2015-12-11 05:18:54 UTC
@Remi, I noticed [1] is just a proposal, and the true fix is [2]. And I found [2] has been merged into the bits I'm using. According to your suggestion, I reviewed `php-m` and opcache appears TWICE. In the customer config, opcache is specified in /etc/php.d/opcache.ini and /etc/php.ini. ie, CG(interned_strings_start) is replaced TWICE and the logic of [2] is broken.

Sorry for this silly bugzilla. It is not a bug, but a configuration issue.
And thank you for your excellent suggestions and insights.

[1] https://bugs.php.net/patch-display.php?bug=65338&patch=zend_interned_strings_shutdown_AV&revision=1374773456
[2] http://git.php.net/?p=php-src.git;a=commit;h=3550f3d0aad6e979e2a6fe3ee40d4fbff168c34b

BTW, is it possible to add a feature to forbid twice-loading?

Comment 10 Remi Collet 2015-12-11 11:54:28 UTC
Very strand... the twice-loading check exists... but only for standard extensions... not for zend_extension.

Upstreal bug open  https://bugs.php.net/71089

Comment 13 Remi Collet 2015-12-11 12:53:02 UTC
Small notice: this is a usual mistake in php configuration, when people blindly follow some internet doc, which explain they have to add the (zend_)extension line in their php.ini, when php.d/foo.ini already provides this line.

And, despite this is documented in php.ini

;;;;
; Note: packaged extension modules are now loaded via the .ini files
; found in the directory /etc/php.d; these are loaded by default.
;;;;

Comment 21 errata-xmlrpc 2016-11-03 21:07:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2598.html


Note You need to log in before you can comment on or make changes to this bug.