Bug 1275276

Created attachment 1086501 [details]
gdb debugging output

Description of problem:

Note: Same site setup on OS 6.5 with Apache 2.2.15 (PHP 5.3.3) was working without any problems (without Segmentation faults)

Segmentation fault triggers in following scenarios:
On heavy traffic site (like our) it rounds about 500-1000 segmentations per day. Which is a lot.

1.Apache reload i.e. during log rotation (systemctl reload httpd)

2. With client requests that are pipelined.
Probably when Apache destroys all of pipelined responses at once (when MAX_REQUESTS_IN_PIPELINE limit reached).
At this moment MAX_REQUESTS_IN_PIPELINE is value set to 5 (cannot be edited). 
So during high traffic on the site those segmentations occurs in batches (when 3-5 requests are destroyed by Apache). 

3. While the client connection is still established and previous PHP request did not completed writing of output buffer and then comes new request from the same client connection with enough output to overwrite buffer.

Same situation with KeepAlive off, opcode cache switched On or Off.

I did some debugging (using gdb) and to me it seems the problem is related to php5 Apache2 handler - file: sapi_apache2.c
See gdb output which show calling perform_idle_server_maintenance(), php_apache_child_shutdown () functions before segmentation. 
Although the real cause was change in the way apache 2.4.x is now cleaning up r->pool.

See discussion in this bug (comments #7 upwards) 
https://bz.apache.org/bugzilla/show_bug.cgi?id=56984 

Like "gmoniker" wrote:
There is now activity going on in the php5 Apache2 handler to solve the segfaults with pipelined client requests by not relying anymore on the destruction of the request pool between two client requests on the same connection.

Thanks in advance for any solution or the information how this issue can be mitigated.


Version-Release number of selected component (if applicable):
php-5.4.16-36.el7_1.x86_64
httpd-2.4.6-31.el7_1.1.x86_64

How reproducible:
Badly, only on heavy traffic sites 


Actual results:
Around 500 - 1000 segmentations per day

Expected results:
Mitigation

Additional info:
---------------------------------
Output from our apache error_log
---------------------------------
[Fri Oct 23 07:28:01.516907 2015] [core:notice] [pid 29922] AH00052: child pid 13232 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:03.519709 2015] [core:notice] [pid 29922] AH00052: child pid 13236 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:06.522733 2015] [core:notice] [pid 29922] AH00052: child pid 16435 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:07.524013 2015] [core:notice] [pid 29922] AH00052: child pid 14616 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:12.530719 2015] [core:notice] [pid 29922] AH00052: child pid 13233 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:14.533456 2015] [core:notice] [pid 29922] AH00052: child pid 13238 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:15.535049 2015] [core:notice] [pid 29922] AH00052: child pid 13235 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:16.536651 2015] [core:notice] [pid 29922] AH00052: child pid 13454 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:17.538077 2015] [core:notice] [pid 29922] AH00052: child pid 13453 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:19.540846 2015] [core:notice] [pid 29922] AH00052: child pid 17805 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:51.575465 2015] [core:notice] [pid 29922] AH00052: child pid 13231 exit signal Segmentation fault (11)


Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Aug 10 2015 07:39:34
Server's Module Magic Number: 20120211:24
Server loaded:  APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

-------------------------
MPM prefork, PHP module
-------------------------
<IfModule prefork.c>
  LoadModule php5_module modules/libphp5.so
  StartServers       8
  MinSpareServers    5
  MaxSpareServers   20
  ServerLimit      256
  MaxClients       256
  MaxRequestsPerChild  0
</IfModule>