Bug 1275276 - After switching to RHEL 7.1 (Apache 2.4.6, PHP 5.4.16) frequent and random Segmentation fault (11)
After switching to RHEL 7.1 (Apache 2.4.6, PHP 5.4.16) frequent and random Se...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: php (Show other bugs)
7.1
x86_64 Linux
unspecified Severity medium
: rc
: ---
Assigned To: Web Stack Team
BaseOS QE - Apps
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-26 08:42 EDT by Josef S.
Modified: 2016-03-09 09:12 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-09 09:12:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
gdb debugging output (3.96 KB, text/plain)
2015-10-26 08:42 EDT, Josef S.
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Apache Bugzilla 56984 None None None Never

  None (edit)
Description Josef S. 2015-10-26 08:42:15 EDT
Created attachment 1086501 [details]
gdb debugging output

Description of problem:

Note: Same site setup on OS 6.5 with Apache 2.2.15 (PHP 5.3.3) was working without any problems (without Segmentation faults)

Segmentation fault triggers in following scenarios:
On heavy traffic site (like our) it rounds about 500-1000 segmentations per day. Which is a lot.

1.Apache reload i.e. during log rotation (systemctl reload httpd)

2. With client requests that are pipelined.
Probably when Apache destroys all of pipelined responses at once (when MAX_REQUESTS_IN_PIPELINE limit reached).
At this moment MAX_REQUESTS_IN_PIPELINE is value set to 5 (cannot be edited). 
So during high traffic on the site those segmentations occurs in batches (when 3-5 requests are destroyed by Apache). 

3. While the client connection is still established and previous PHP request did not completed writing of output buffer and then comes new request from the same client connection with enough output to overwrite buffer.

Same situation with KeepAlive off, opcode cache switched On or Off.

I did some debugging (using gdb) and to me it seems the problem is related to php5 Apache2 handler - file: sapi_apache2.c
See gdb output which show calling perform_idle_server_maintenance(), php_apache_child_shutdown () functions before segmentation. 
Although the real cause was change in the way apache 2.4.x is now cleaning up r->pool.

See discussion in this bug (comments #7 upwards) 
https://bz.apache.org/bugzilla/show_bug.cgi?id=56984 

Like "gmoniker" wrote:
There is now activity going on in the php5 Apache2 handler to solve the segfaults with pipelined client requests by not relying anymore on the destruction of the request pool between two client requests on the same connection.

Thanks in advance for any solution or the information how this issue can be mitigated.


Version-Release number of selected component (if applicable):
php-5.4.16-36.el7_1.x86_64
httpd-2.4.6-31.el7_1.1.x86_64

How reproducible:
Badly, only on heavy traffic sites 


Actual results:
Around 500 - 1000 segmentations per day

Expected results:
Mitigation

Additional info:
---------------------------------
Output from our apache error_log
---------------------------------
[Fri Oct 23 07:28:01.516907 2015] [core:notice] [pid 29922] AH00052: child pid 13232 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:03.519709 2015] [core:notice] [pid 29922] AH00052: child pid 13236 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:06.522733 2015] [core:notice] [pid 29922] AH00052: child pid 16435 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:07.524013 2015] [core:notice] [pid 29922] AH00052: child pid 14616 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:12.530719 2015] [core:notice] [pid 29922] AH00052: child pid 13233 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:14.533456 2015] [core:notice] [pid 29922] AH00052: child pid 13238 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:15.535049 2015] [core:notice] [pid 29922] AH00052: child pid 13235 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:16.536651 2015] [core:notice] [pid 29922] AH00052: child pid 13454 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:17.538077 2015] [core:notice] [pid 29922] AH00052: child pid 13453 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:19.540846 2015] [core:notice] [pid 29922] AH00052: child pid 17805 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:51.575465 2015] [core:notice] [pid 29922] AH00052: child pid 13231 exit signal Segmentation fault (11)


Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Aug 10 2015 07:39:34
Server's Module Magic Number: 20120211:24
Server loaded:  APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

-------------------------
MPM prefork, PHP module
-------------------------
<IfModule prefork.c>
  LoadModule php5_module modules/libphp5.so
  StartServers       8
  MinSpareServers    5
  MaxSpareServers   20
  ServerLimit      256
  MaxClients       256
  MaxRequestsPerChild  0
</IfModule>
Comment 3 Josef S. 2016-01-21 04:20:02 EST
Update:

On December 18th of 2015 we applied the latest 7.2 release of the operating system (about 500 packages were updated in the upgrade process).
Immediately - after all of the packages were installed and the machine was restarted - segmentation stopped. 

Ever since (up to this day) not a single segmentation error was detected in our apache error_log files.
So it's safe to say the segmentation issue was mitigated.

Message to other users - upgrading to 7.2. can solve this issue.

Because 7.2. release did contain any new nor changed version of the php package I assume that something must been changed in the httpd package itself. 
7.2. release contains httpd-2.4.6-40.el7 package (previous version httpd-2.4.6-31.el7 - 7.1. release - was giving segmentations).

We will appreciate if you could briefly specify what changes in the httpd-2.4.6-40.el7 package helped to resolve segmentation issue.   
Otherwise we have been lucky that our issue was mitigated with the 7.2. release of operating system.

Best regards,

Josef
Comment 4 Joe Orton 2016-03-09 09:12:03 EST
Good to hear the issue is fixed!  It is probably the OpenLDAP issue in bug 1158005 from the backtrace.

I'd recommend contacting Red Hat support if you want assistance with an issue in production in the future, we can better assist you diagnosing & solving issues that way.

Note You need to log in before you can comment on or make changes to this bug.