RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1275276 - After switching to RHEL 7.1 (Apache 2.4.6, PHP 5.4.16) frequent and random Segmentation fault (11)
Summary: After switching to RHEL 7.1 (Apache 2.4.6, PHP 5.4.16) frequent and random Se...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: php
Version: 7.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Remi Collet
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-26 12:42 UTC by Josef S.
Modified: 2021-01-14 09:26 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-09 14:12:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
gdb debugging output (3.96 KB, text/plain)
2015-10-26 12:42 UTC, Josef S.
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Apache Bugzilla 56984 0 None None None Never

Description Josef S. 2015-10-26 12:42:15 UTC
Created attachment 1086501 [details]
gdb debugging output

Description of problem:

Note: Same site setup on OS 6.5 with Apache 2.2.15 (PHP 5.3.3) was working without any problems (without Segmentation faults)

Segmentation fault triggers in following scenarios:
On heavy traffic site (like our) it rounds about 500-1000 segmentations per day. Which is a lot.

1.Apache reload i.e. during log rotation (systemctl reload httpd)

2. With client requests that are pipelined.
Probably when Apache destroys all of pipelined responses at once (when MAX_REQUESTS_IN_PIPELINE limit reached).
At this moment MAX_REQUESTS_IN_PIPELINE is value set to 5 (cannot be edited). 
So during high traffic on the site those segmentations occurs in batches (when 3-5 requests are destroyed by Apache). 

3. While the client connection is still established and previous PHP request did not completed writing of output buffer and then comes new request from the same client connection with enough output to overwrite buffer.

Same situation with KeepAlive off, opcode cache switched On or Off.

I did some debugging (using gdb) and to me it seems the problem is related to php5 Apache2 handler - file: sapi_apache2.c
See gdb output which show calling perform_idle_server_maintenance(), php_apache_child_shutdown () functions before segmentation. 
Although the real cause was change in the way apache 2.4.x is now cleaning up r->pool.

See discussion in this bug (comments #7 upwards) 
https://bz.apache.org/bugzilla/show_bug.cgi?id=56984 

Like "gmoniker" wrote:
There is now activity going on in the php5 Apache2 handler to solve the segfaults with pipelined client requests by not relying anymore on the destruction of the request pool between two client requests on the same connection.

Thanks in advance for any solution or the information how this issue can be mitigated.


Version-Release number of selected component (if applicable):
php-5.4.16-36.el7_1.x86_64
httpd-2.4.6-31.el7_1.1.x86_64

How reproducible:
Badly, only on heavy traffic sites 


Actual results:
Around 500 - 1000 segmentations per day

Expected results:
Mitigation

Additional info:
---------------------------------
Output from our apache error_log
---------------------------------
[Fri Oct 23 07:28:01.516907 2015] [core:notice] [pid 29922] AH00052: child pid 13232 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:03.519709 2015] [core:notice] [pid 29922] AH00052: child pid 13236 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:06.522733 2015] [core:notice] [pid 29922] AH00052: child pid 16435 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:07.524013 2015] [core:notice] [pid 29922] AH00052: child pid 14616 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:12.530719 2015] [core:notice] [pid 29922] AH00052: child pid 13233 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:14.533456 2015] [core:notice] [pid 29922] AH00052: child pid 13238 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:15.535049 2015] [core:notice] [pid 29922] AH00052: child pid 13235 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:16.536651 2015] [core:notice] [pid 29922] AH00052: child pid 13454 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:17.538077 2015] [core:notice] [pid 29922] AH00052: child pid 13453 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:19.540846 2015] [core:notice] [pid 29922] AH00052: child pid 17805 exit signal Segmentation fault (11)
[Fri Oct 23 07:28:51.575465 2015] [core:notice] [pid 29922] AH00052: child pid 13231 exit signal Segmentation fault (11)


Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built:   Aug 10 2015 07:39:34
Server's Module Magic Number: 20120211:24
Server loaded:  APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

-------------------------
MPM prefork, PHP module
-------------------------
<IfModule prefork.c>
  LoadModule php5_module modules/libphp5.so
  StartServers       8
  MinSpareServers    5
  MaxSpareServers   20
  ServerLimit      256
  MaxClients       256
  MaxRequestsPerChild  0
</IfModule>

Comment 3 Josef S. 2016-01-21 09:20:02 UTC
Update:

On December 18th of 2015 we applied the latest 7.2 release of the operating system (about 500 packages were updated in the upgrade process).
Immediately - after all of the packages were installed and the machine was restarted - segmentation stopped. 

Ever since (up to this day) not a single segmentation error was detected in our apache error_log files.
So it's safe to say the segmentation issue was mitigated.

Message to other users - upgrading to 7.2. can solve this issue.

Because 7.2. release did contain any new nor changed version of the php package I assume that something must been changed in the httpd package itself. 
7.2. release contains httpd-2.4.6-40.el7 package (previous version httpd-2.4.6-31.el7 - 7.1. release - was giving segmentations).

We will appreciate if you could briefly specify what changes in the httpd-2.4.6-40.el7 package helped to resolve segmentation issue.   
Otherwise we have been lucky that our issue was mitigated with the 7.2. release of operating system.

Best regards,

Josef

Comment 4 Joe Orton 2016-03-09 14:12:03 UTC
Good to hear the issue is fixed!  It is probably the OpenLDAP issue in bug 1158005 from the backtrace.

I'd recommend contacting Red Hat support if you want assistance with an issue in production in the future, we can better assist you diagnosing & solving issues that way.


Note You need to log in before you can comment on or make changes to this bug.