Description of problem: This bug is probably caused by Zend OPCache PECL. When httpd receives -NOHUP signal (which should reload the Apache), the PHP 5.4 (mod_php) segfaults and brings down the whole thing. This happens only when Zend OPCache PECL is enabled. Backtrace: > #0 0x00007f6d682016e0 in __write_nocancel () from /lib64/libpthread.so.0 > #1 0x00007f6d684248bf in apr_file_write () from /usr/lib64/libapr-1.so.0 > #2 0x00007f6d684252fb in apr_file_write_full () from /usr/lib64/libapr-1.so.0 > #3 0x00007f6d6995439e in log_error_core (file=0x7f6d6996b6c8 "/builddir/build/BUILD/httpd-2.2.15/server/mpm_common.c", line=1229, level=5, status=0, s=0x7f6d6aebf8b0, c=<value optimized out>, r=0x0, pool=0x0, > fmt=0x7f6d6996b700 "seg fault or similar nasty error detected in the parent process", args=0x7fff07debf10) at /usr/src/debug/httpd-2.2.15/server/log.c:696 > #4 0x00007f6d69954926 in ap_log_error (file=<value optimized out>, line=<value optimized out>, level=<value optimized out>, status=<value optimized out>, s=<value optimized out>, fmt=<value optimized out>) > at /usr/src/debug/httpd-2.2.15/server/log.c:715 > #5 0x00007f6d69958dab in sig_coredump (sig=11) at /usr/src/debug/httpd-2.2.15/server/mpm_common.c:1229 > #6 <signal handler called> > #7 0x00007f6d67eda53c in free () from /lib64/libc.so.6 > #8 0x00007f6d6737990a in php_module_shutdown () at /usr/src/debug/php-5.4.16/main/main.c:2367 > #9 0x00007f6d673799c9 in php_module_shutdown_wrapper (sapi_globals=<value optimized out>) at /usr/src/debug/php-5.4.16/main/main.c:2335 > #10 0x00007f6d67481bb1 in php_apache_server_shutdown (tmp=<value optimized out>) at /usr/src/debug/php-5.4.16/sapi/apache2handler/sapi_apache2.c:388 > #11 0x00007f6d68428b9e in apr_pool_clear () from /usr/lib64/libapr-1.so.0 > #12 0x00007f6d6993b865 in main (argc=11, argv=0x7fff07dec7e8) at /usr/src/debug/httpd-2.2.15/server/main.c:713 Version-Release number of selected component (if applicable): > httpd-2.2.15-39.el6.x86_64 > php54-1.1-5.el6.x86_64 > php54-php-5.4.16-22.el6.x86_64 > php54-php-pecl-zendopcache-7.0.2-4.el6.x86_64 The following might be related: - http://www.electrictoolbox.com/seg-fault-or-similar-nasty-error/ - https://bugs.php.net/bug.php?id=27899 - https://bugs.php.net/bug.php?id=27810 - http://forums.zend.com/viewtopic.php?f=57&t=1242 How reproducible: always Steps to Reproduce: 1. Enable PHP 5.4 + Apache (via mod_php) 2. Enable Zend OPCache PECL module 3. Run httpd (make sure PHP loads the OPCache) 4. Send "NOHUP" signal to the httpd process Actual results: Segmentation fault Logs: > [Fri Dec 12 21:26:13 2014] [notice] SIGHUP received. Attempting to restart > [Fri Dec 12 21:26:13 2014] [notice] seg fault or similar nasty error detected in the parent process Expected results: > It should reload the Apache without any error.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/480d3fe4d4a7743db012ca41412d1a86889e9a3e Fix bug 1173796 https://bugzilla.redhat.com/show_bug.cgi?id=1173796
Sorry... cannot reproduce. A full list of all php extensions and httpd modules installed will be useful. Can you test with opcache.fast_shutdown=0 (in /opt/rh/php54/root/etc/php.d/opcache.ini)
> ls /opt/rh/php54/root/etc/php.d/ > apc.ini dom.ini imagick.ini ldap.ini memcache.ini mysqlnd_mysqli.ini opcache.ini pdo_pgsql.ini phar.ini sqlite3.ini sysvshm.ini xmlreader.ini zip.ini > bcmath.ini fileinfo.ini intl.ini mbstring.ini mongo.ini mysqlnd_mysql.ini pdo.ini pdo_sqlite.ini posix.ini sysvmsg.ini wddx.ini xmlwriter.ini > curl.ini gd.ini json.ini mcrypt.ini mysqlnd.ini opcache-default.blacklist pdo_mysqlnd.ini pgsql.ini soap.ini sysvsem.ini xdebug.ini xsl.ini > # php -m > [PHP Modules] > apc > bcmath > bz2 > calendar > Core > ctype > curl > date > dom > ereg > exif > fileinfo > filter > ftp > gd > gettext > gmp > hash > iconv > imagick > imap > intl > json > libxml > mbstring > mcrypt > memcache > memcached > mongo > mysql > mysqli > openssl > pcntl > pcre > PDO > pdo_mysql > pdo_pgsql > pdo_sqlite > pgsql > Phar > posix > readline > Reflection > session > shmop > SimpleXML > soap > sockets > SPL > sqlite3 > standard > sysvmsg > sysvsem > sysvshm > tidy > tokenizer > wddx > xdebug > xml > xmlreader > xmlwriter > xsl > zip > zlib > > [Zend Modules] > Xdebug
Disabling opcache.fast_shutdown didn't help. > php -r "phpinfo();" | grep opcache > opcache.blacklist_filename => /opt/rh/php54/root/etc/php.d/opcache*.blacklist => /opt/rh/php54/root/etc/php.d/opcache*.blacklist > opcache.consistency_checks => 0 => 0 > opcache.dups_fix => Off => Off > opcache.enable => On => On > opcache.enable_cli => Off => Off > opcache.enable_file_override => Off => Off > opcache.error_log => no value => no value > opcache.fast_shutdown => 0 => 0 > opcache.force_restart_timeout => 180 => 180 > opcache.inherited_hack => On => On > opcache.interned_strings_buffer => 8 => 8 > opcache.load_comments => 1 => 1 > opcache.log_verbosity_level => 1 => 1 > opcache.max_accelerated_files => 4000 => 4000 > opcache.max_file_size => 1M => 1M > opcache.max_wasted_percentage => 5 => 5 > opcache.memory_consumption => 64M => 64M > opcache.optimization_level => 0xFFFFFFFF => 0xFFFFFFFF > opcache.preferred_memory_model => no value => no value > opcache.protect_memory => 0 => 0 > opcache.revalidate_freq => 2 => 2 > opcache.revalidate_path => Off => Off > opcache.save_comments => 1 => 1 > opcache.use_cwd => On => On > opcache.validate_timestamps => On => On Still hitting the same SIGSEGV.
# yum info php54-* httpd-* mod_* | awk '/Name/ { printf "%s-",$3 } /Version/ { printf "%s-",$3 } /Release/ { printf "%s\n",$3 }' | sort httpd-2.2.15-39.el6 httpd-debuginfo-2.2.15-39.el6 httpd-devel-2.2.15-39.el6 httpd-devel-2.2.15-39.el6 httpd-itk-2.2.22-7.el6 httpd-manual-2.2.15-39.el6 httpd-tools-2.2.15-39.el6 mod_auth_cas-1.0.9.1-1.el6 mod_auth_kerb-5.4-13.el6 mod_auth_mellon-0.8.0-3.el6_6 mod_auth_mysql-3.0.0-11.el6_0.1 mod_auth_ntlm_winbind-0.0.0-0.10.20070129svn713.el6 mod_authnz_external-3.2.6-1.el6 mod_authnz_pam-0.9.2-1.el6 mod_auth_openid-0.7-2.el6 mod_auth_pam-1.1.1-9.el6 mod_auth_pgsql-2.0.3-10.1.el6 mod_auth_shadow-2.3-2.el6 mod_auth_token-1.0.5-2.el6 mod_auth_xradius-0.4.6-16.el6 mod_authz_ldap-0.26-16.el6 mod_autoindex_mb-2.18.1-1.el6 mod_bw-0.8-4.el6 mod_cluster-1.2.6-3.1.el6 mod_cluster-1.2.9-1.Final_redhat_1.1.ep6.el6 mod_cluster-demo-1.2.9-1.Final_redhat_1.1.ep6.el6 mod_cluster-jbossas-1.0.10-4.1.GA_CP02_patch01.ep5.el6 mod_cluster-jbossweb2-1.0.10-4.1.GA_CP02_patch01.ep5.el6 mod_cluster-native-1.2.9-3.Final_redhat_2.ep6.el6 mod_cluster-tomcat6-1.2.9-1.Final_redhat_1.1.ep6.el6 mod_cluster-tomcat7-1.2.9-1.Final_redhat_1.1.ep6.el6 mod_dav_svn-1.6.11-10.el6_5 mod_dnssd-0.6-2.el6 mod_evasive-1.10.1-10.el6 mod_extract_forwarded-2.0.2-8.el6 mod_fcgid-2.3.9-1.el6 mod_flvx-0-0.1.20100525git.el6 mod_form-0.1-1.20131204svn145.el6 mod_form-0.1-1.20131204svn145.el6 mod_form-devel-0.1-1.20131204svn145.el6 mod_form-devel-0.1-1.20131204svn145.el6 mod_geoip-1.2.7-1.el6 mod_intercept_form_submit-0.9.7-1.el6 mod_jk-ap20-1.2.31-1.1.2.ep5.el6 mod_jk-ap22-1.2.40-2.redhat_1.ep6.el6 mod_jk-manual-1.2.40-2.redhat_1.ep6.el6 mod_limitipconn-0.23-3.el6 mod_log_post-0.1.0-2.el6 mod_lookup_identity-0.9.2-1.el6 mod_mirrorbrain-2.18.1-1.el6 mod_nss-1.0.8-21.el6 mod_passenger-3.0.21-12.el6oso mod_perl-2.0.4-11.el6_5 mod_perl-devel-2.0.4-11.el6_5 mod_perl-devel-2.0.4-11.el6_5 mod_proxy_html-3.1.2-6.el6 mod_pubcookie-3.3.4a-3.el6.1 mod_pubcookie-server-3.3.4a-3.el6.1 mod_python-3.3.1-16.el6 mod_qos-10.24-1.el6 mod_revocator-1.0.3-10.el6 mod_revocator-1.0.3-10.el6 mod_ridealong-3.0.13-3.el6_3 mod_rt-2.4.1-6.GA.ep6.el6 mod_security-2.7.3-3.el6 mod_security_crs-2.2.6-3.el6 mod_security_crs-extras-2.2.6-3.el6 mod_snmp-2.4.1-13.GA.ep6.el6 mod_speedycgi-2.22-8.el6 mod_ssl-2.2.15-39.el6 mod_vhost_choke-0.1-2.el6_3 mod_wsgi-3.2-7.el6 mod_xsendfile-0.12-3.el6 php54-1.1-5.el6 php54-apc-panel-3.1.15-0.4.20130912.el6 php54-php-5.4.16-22.el6 php54-php-bcmath-5.4.16-22.el6 php54-php-cli-5.4.16-22.el6 php54-php-common-5.4.16-22.el6 php54-php-dba-5.4.16-22.el6 php54-php-debuginfo-5.4.16-22.el6 php54-php-devel-5.4.16-22.el6 php54-php-enchant-5.4.16-22.el6 php54-php-fpm-5.4.16-22.el6 php54-php-gd-5.4.16-22.el6 php54-php-imap-5.4.16-22.el6 php54-php-interbase-5.4.16-3.el6oso php54-php-intl-5.4.16-22.el6 php54-php-ldap-5.4.16-22.el6 php54-php-mbstring-5.4.16-22.el6 php54-php-mcrypt-5.4.16-3.el6oso php54-php-mssql-5.4.16-3.el6oso php54-php-mysqlnd-5.4.16-22.el6 php54-php-odbc-5.4.16-22.el6 php54-php-pdo-5.4.16-22.el6 php54-php-pear-1.9.4-10.el6 php54-php-pecl-apc-3.1.15-0.4.20130912.el6 php54-php-pecl-apc-debuginfo-3.1.15-0.4.20130912.el6 php54-php-pecl-apc-devel-3.1.15-0.4.20130912.el6 php54-php-pecl-imagick-3.1.2-1.el6oso php54-php-pecl-memcache-3.0.8-3.el6 php54-php-pecl-memcache-debuginfo-3.0.8-3.el6 php54-php-pecl-mongo-1.4.5-1.el6oso php54-php-pecl-xdebug-2.2.3-3.el6oso php54-php-pecl-zendopcache-7.0.2-4.el6 php54-php-pecl-zendopcache-debuginfo-7.0.2-4.el6 php54-php-pgsql-5.4.16-22.el6 php54-php-process-5.4.16-22.el6 php54-php-pspell-5.4.16-22.el6 php54-php-recode-5.4.16-22.el6 php54-php-snmp-5.4.16-22.el6 php54-php-soap-5.4.16-22.el6 php54-php-tidy-5.4.16-22.el6 php54-php-xcache-3.1.0-3.el6oso php54-php-xml-5.4.16-22.el6 php54-php-xmlrpc-5.4.16-22.el6 php54-runtime-1.1-5.el6 php54-scldevel-1.1-5.el6 php54-xcache-admin-3.1.0-3.el6oso
With all available modules in RHEL, I cannot reporoduce. Which is very strange, you have 3 opcode cache installed... You should not even be able to install them, as php54-php-pecl-zendopcache-7.0.2-4.el6.x86_64 conflicts with php54-php-xcache Can you try with a single opcode cache ?
@Remi, seems like my previous command to get all the installed RPMs was wrong. > Package(s) php54-php-xcache available, but not installed. For PHP 5.4 APC, we strictly have > apc.enable_opcode_cache=0 So, the Zend OPCache should be the only opcode cache used..
# yum list php54-* httpd-* mod_* httpd.x86_64-2.2.15-39.el6 httpd-debuginfo.x86_64-2.2.15-39.el6 httpd-devel.x86_64-2.2.15-39.el6 httpd-tools.x86_64-2.2.15-39.el6 mod_cluster.noarch-1.2.9-1.Final_redhat_1.1.ep6.el6 mod_passenger.x86_64-1:3.0.21-12.el6oso mod_perl.x86_642.0.4-11.el6_5 mod_security.x86_642.7.3-3.el6 mod_ssl.x86_64-1:2.2.15-39.el6 mod_wsgi.x86_643.2-7.el6 php54.x86_64-1.1-5.el6 php54-php.x86_64-5.4.16-22.el6 php54-php-bcmath.x86_645.4.16-22.el6 php54-php-cli.x86_64-5.4.16-22.el6 php54-php-common.x86_645.4.16-22.el6 php54-php-debuginfo.x86_64-5.4.16-22.el6 php54-php-devel.x86_64-5.4.16-22.el6 php54-php-fpm.x86_64-5.4.16-22.el6 php54-php-gd.x86_645.4.16-22.el6 php54-php-intl.x86_645.4.16-22.el6 php54-php-ldap.x86_645.4.16-22.el6 php54-php-mbstring.x86_645.4.16-22.el6 php54-php-mcrypt.x86_645.4.16-3.el6oso php54-php-mysqlnd.x86_64-5.4.16-22.el6 php54-php-pdo.x86_64-5.4.16-22.el6 php54-php-pear.noarch1:1.9.4-10.el6 php54-php-pecl-apc.x86_643.1.15-0.4.20130912.el6 php54-php-pecl-apc-debuginfo.x86_643.1.15-0.4.20130912.el6 php54-php-pecl-apc-devel.x86_643.1.15-0.4.20130912.el6 php54-php-pecl-imagick.x86_643.1.2-1.el6oso php54-php-pecl-memcache.x86_64-3.0.8-3.el6 php54-php-pecl-memcache-debuginfo.x86_64-3.0.8-3.el6 php54-php-pecl-mongo.x86_641.4.5-1.el6oso php54-php-pecl-xdebug.x86_64-2.2.3-3.el6oso php54-php-pecl-zendopcache.x86_647.0.2-4.el6 php54-php-pecl-zendopcache-debuginfo.x86_647.0.2-4.el6 php54-php-pgsql.x86_64-5.4.16-22.el6 php54-php-process.x86_64-5.4.16-22.el6 php54-php-soap.x86_645.4.16-22.el6 php54-php-xml.x86_64-5.4.16-22.el6 php54-runtime.x86_64-1.1-5.el6 php54-scldevel.x86_641.1-5.el6 I'm sorry for providing wrong data previously.
cat php/configuration/etc/php.d/openshift.ini | grep -e apc -e opcache ; Enable apc extension module extension = apc.so ; See http://www.php.net/manual/en/apc.configuration.php apc.enable_opcode_cache=0 apc.enabled=1 apc.shm_size=64M apc.shm_strings_buffer=8M ;apc.num_files_hint=1000 ;apc.user_entries_hint=4096 ;apc.ttl=0 ;apc.use_request_time=1 ;apc.user_ttl=0 ;apc.gc_ttl=3600 ; apc.filters so that files are only cached if matched by a positive filter. ;apc.cache_by_default=1 ;apc.filters= apc.mmap_file_mask=/tmp/apc.XXXXXX ;apc.file_update_protection=2 ;apc.enable_cli=0 ;apc.max_file_size=1M apc.stat=0 ;apc.stat_ctime=0 ;apc.canonicalize=1 ;apc.write_lock=1 ;apc.report_autofilter=0 ;apc.rfc1867=0 ;apc.rfc1867_prefix =upload_ ;apc.rfc1867_name=APC_UPLOAD_PROGRESS ;apc.rfc1867_freq=0 ;apc.rfc1867_ttl=3600 ;apc.coredump_unmap=0 ;apc.file_md5=0 ;apc.preload_path= zend_extension=/opt/rh/php54/root/usr/lib64/php/modules/opcache.so opcache.enable=1 ;opcache.enable_cli=0 opcache.memory_consumption=64M opcache.interned_strings_buffer=8 opcache.max_accelerated_files=4000 ;opcache.max_wasted_percentage=5 ;opcache.use_cwd=1 ;opcache.validate_timestamps=1 ;opcache.revalidate_freq=2 ;opcache.revalidate_path=0 ;opcache.save_comments=1 ;opcache.load_comments=1 opcache.fast_shutdown=1 ;opcache.enable_file_override=0 ;opcache.optimization_level=0xffffffff ;opcache.inherited_hack=1 ;opcache.dups_fix=0 opcache.blacklist_filename=/opt/rh/php54/root/etc/php.d/opcache*.blacklist opcache.max_file_size=1M ;opcache.consistency_checks=0 ;opcache.force_restart_timeout=180 ;opcache.error_log= ;opcache.log_verbosity_level=1 ;opcache.preferred_memory_model= ;opcache.protect_memory=0
From above: apc.shm_strings_buffer=8M opcache.interned_strings_buffer=8 This is known to create segfault. Only 1 extension can use interned strings.
https://github.com/openshift/origin-server/pull/6023 Changing > apc.shm_strings_buffer=0 seems to fix the issue. Thanks for suggestion, Remi. The warning message wasn't in the original apc.ini, as provided by an older RHSCL 1.0. > WARNING: only enable this buffer if you don't use Zend Opcache
> The warning message wasn't in the original apc.ini, as provided by an older RHSCL 1.0. Yes, this issue was discovered later, during QA of opcache for RHSCL 1.1, see bug #1052178
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/f3080260e0fb4095d0f191500003bb351e0074df Fix bug 1173796 https://bugzilla.redhat.com/show_bug.cgi?id=1173796
Verified with devenv_5366 Reload apache without segfaults error when enabled opcache and apc user cache for php-5.4 application. > php -m [PHP Modules] apc bcmath bz2 calendar Core ctype curl date dom ereg exif fileinfo filter ftp gd gettext gmp hash iconv imagick intl json ldap libxml mbstring mcrypt memcache mhash mongo mysql mysqli mysqlnd openssl pcntl pcre PDO pdo_mysql pdo_pgsql pdo_sqlite pgsql Phar posix readline Reflection session shmop SimpleXML soap sockets SPL sqlite3 standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zip zlib [Zend Modules] Zend OPcache > kill -HUP <httpd.pid> > tailf app-root/logs/php.log [Thu Dec 18 02:59:16 2014] [notice] Graceful restart requested, doing restart - - - [18/Dec/2014:02:59:16 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (Red Hat) (internal dummy connection)" [Thu Dec 18 02:59:16 2014] [notice] Digest: generating secret for digest authentication ... [Thu Dec 18 02:59:16 2014] [notice] Digest: done [Thu Dec 18 02:59:16 2014] [notice] Apache/2.2.15 (Unix) configured -- resuming normal operations Mark bug as verified.