Without a repro case this type of php crash is hard to track down.

This looks like a refcount issue, as Joe said, very hard to track because the issue raise the segfault later, during shutdown.

Notice: as this occurs in shutdown, the request have been served.

A list of installed extensions may help ("rpm -qa php\* | sort" and "php -m")

Could also be related to gc.

A possible workaround is to disable gc (zend.enable_gc=0), but this may require to increase memory_limit.

Joe and Remi, I really appreciate your prompt response. I will try to gather up info to reproduce the issue.

@Joe, Remi, I could reproduce the issue and found the cause, the workaround, and how to fix.

The issue is reproduced when stopping httpd/php with *opcache*. Installing the php-pecl-zendopcache package, you can reproduce the issue simply running `systemcrl start httpd` then `systemctl stop httpd`.

The issue occurs when freeing CG(interned_strings_start) in zend_interned_strings_dtor(). The code is as follows:-

/usr/src/debug/php-5.4.16/Zend/zend_string.c:
------------------------------------------------------
void zend_interned_strings_dtor(TSRMLS_D)
{
#ifndef ZTS
#if ZEND_DEBUG_INTERNED_STRINGS
	mprotect(CG(interned_strings_start), CG(interned_strings_end) - CG(interned_strings_start), PROT_WRITE | PROT_READ);
#endif
	free(CG(interned_strings).arBuckets);
	free(CG(interned_strings_start)); <<===HERE
#endif
}
------------------------------------------------------

backtrace:
------------------------------------------------------
#0  zend_interned_strings_dtor () at /usr/src/debug/php-5.4.16/Zend/zend_string.c:78
#1  0x00007f4d8988f47b in php_module_shutdown () at /usr/src/debug/php-5.4.16/main/main.c:2367
#2  0x00007f4d8988f539 in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at /usr/src/debug/php-5.4.16/main/main.c:2335
#3  0x00007f4d8999ab21 in php_apache_child_shutdown (tmp=<optimized out>)
    at /usr/src/debug/php-5.4.16/sapi/apache2handler/sapi_apache2.c:398
#4  0x00007f4d976f61ae in apr_pool_destroy () from /lib64/libapr-1.so.0
#5  0x00007f4d8e41823c in clean_child_exit (code=code@entry=0) at prefork.c:221
#6  0x00007f4d8e4186e7 in child_main (child_num_arg=child_num_arg@entry=0) at prefork.c:728
#7  0x00007f4d8e418a55 in make_child (s=0x7f4d99227340, slot=slot@entry=0) at prefork.c:810
#8  0x00007f4d8e418ab6 in startup_children (number_to_start=1) at prefork.c:828
#9  0x00007f4d8e4197c0 in prefork_run (_pconf=<optimized out>, plog=0x7f4d9922b378, s=0x7f4d99227340) at prefork.c:986
#10 0x00007f4d98a2350e in ap_run_mpm (pconf=0x7f4d991fe158, plog=0x7f4d9922b378, s=0x7f4d99227340) at mpm_common.c:96
#11 0x00007f4d98a1cb36 in main (argc=2, argv=0x7fffb4909928) at main.c:777
------------------------------------------------------

If opcache is not installed, CG(interned_strings_start) points a memory chunk which is normally malloc-ed. Therefore, freeing it works fine. If opcache is installed, CG(interned_strings_start) is replaced with a pointer to a "zend_shared_alloc"-ed memory chunk in zend_accel_init_shm() in /usr/src/debug/php-pecl-zendopcache-7.0.5/NTS/ZendAccelerator.c. Since the chunk is not a malloc-ed one, freeing it leads to ABRT.

The workaround is to set 0 to opcache.interned_strings_buffer in /etc/php.d/opcache.ini. Since it disables the "interning strings" feature, it may affect memory usage and through-put, but it is better than disabling opcache itself.

The fix was provided in [1] which is applied into the php core (not opcache). I found the same issue was filed in [2]. I know the php-pecl-zendopcache package is provided in EPEL repository and not fully supported as of now. However, I think we should merge the fix, because the fix is small and surely harmless, testing it is very easy (just stop httpd), and it is safe for future when the opcache is fully supported, and to top it off, our customer needs opcache. Please merge it in the next build.

[1] https://bugs.php.net/patch-display.php?bug=65338&patch=zend_interned_strings_shutdown_AV&revision=1374773456
[2] https://bugs.php.net/bug.php?id=65338

Reproduced using:

php-5.4.16-36.el7_1.x86_64
php-pecl-zendopcache-7.0.5-1.el7.x86_64

@Hisanobu: great thanks for your analysis.

Strangely, the patch which is supposed to fix upstream bug 65338 is part opcache (7.0.3-dev), so is present in 7.0.5.

I will dig further on this;

Some testing:

Startup:
- zend_interned_string_init => alloc
- zend_accel_init_shm => save initial value + reallocation

Shutdown:
- accel_shutdown => restore initial value
- zend_interned_string_dtor => free

So from my test, cannot raise the issue (but with 5.5, will try the same with 5.4.16 asap)

@Hisanobu can you please ensure there is no other extension ? (ex: APC... which also play with interned strings...): php -m

@Remi, I noticed [1] is just a proposal, and the true fix is [2]. And I found [2] has been merged into the bits I'm using. According to your suggestion, I reviewed `php-m` and opcache appears TWICE. In the customer config, opcache is specified in /etc/php.d/opcache.ini and /etc/php.ini. ie, CG(interned_strings_start) is replaced TWICE and the logic of [2] is broken.

Sorry for this silly bugzilla. It is not a bug, but a configuration issue.
And thank you for your excellent suggestions and insights.

[1] https://bugs.php.net/patch-display.php?bug=65338&patch=zend_interned_strings_shutdown_AV&revision=1374773456
[2] http://git.php.net/?p=php-src.git;a=commit;h=3550f3d0aad6e979e2a6fe3ee40d4fbff168c34b

BTW, is it possible to add a feature to forbid twice-loading?

Very strand... the twice-loading check exists... but only for standard extensions... not for zend_extension.

Upstreal bug open  https://bugs.php.net/71089

And upstream patch http://git.php.net/?p=php-src.git;a=patch;h=410eacc1a9b50ec3cb6c5fc0ff252516d0c0a4f1

Small notice: this is a usual mistake in php configuration, when people blindly follow some internet doc, which explain they have to add the (zend_)extension line in their php.ini, when php.d/foo.ini already provides this line.

And, despite this is documented in php.ini

;;;;
; Note: packaged extension modules are now loaded via the .ini files
; found in the directory /etc/php.d; these are loaded by default.
;;;;

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2598.html