Bug 1268352 - Tomcat 6 NIO connector memory leak
Summary: Tomcat 6 NIO connector memory leak
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: tomcat6
Version: 6.7
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: Coty Sutherland
QA Contact: Michal Karm Babacek
Marc Muehlfeld
URL:
Whiteboard:
: 1272985 (view as bug list)
Depends On:
Blocks: 1172231 1275725 1272985 1301646
TreeView+ depends on / blocked
 
Reported: 2015-10-02 15:41 UTC by Aaron Ogburn
Modified: 2019-10-10 10:17 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The Tomcat 6 NIO connector does not leak memory anymore Previously, a memory leak sometimes occurred when using the Tomcat 6 Non-blocking I/O (NIO) connector. This update ensures that Tomcat 6 removes processors from the RequestGroupInfo list and returns them to the recycledProcessors queue. As a result, the NIO connector no longer leaks memory.
Clone Of:
: 1272985 1301646 (view as bug list)
Environment:
Last Closed: 2016-05-11 00:05:40 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0870 normal SHIPPED_LIVE tomcat6 bug fix and enhancement update 2016-05-10 22:47:20 UTC
Red Hat Knowledge Base (Solution) 1978913 None None None 2016-02-11 17:40:34 UTC

Description Aaron Ogburn 2015-10-02 15:41:24 UTC
Description of problem:

Tomcat NIO connector has a very obvious memory leak introduced as a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1031327.  The NIO connector does not properly offer processors back to the recycledProcessors queue.  Thus, the processors are not cleaned up from the o.a.c.RequestGroupInfo's processors list, leading to an easy OOME as Tomcat keeps creating more and more processors as they are never recycled and indefinitely kept in the RequestGroupInfo processors list.

It looks like NIO leaks such as this existed in some capacity before bz-1031327, but this made it even more evident by removing the most common recycledProccessors.offer call.


Version-Release number of selected component (if applicable):

6.0.24-90

How reproducible:

Very.


Steps to Reproduce:
1. Configure your server.xml to use NIO:

<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"

2. Make repeated requests to the NIO connector.  For instance, the following script to do so easily:

#!/bin/bash
N_REQ=0
TOTAL_REQUESTS=100000
while [ $((N_REQ)) -lt $((TOTAL_REQUESTS)) ]
do
curl "http://localhost:8080/sample/hello" > /dev/null 2>&1
N_REQ=$((N_REQ +1))
echo $N_REQ
done

3. Note that heap usage continues to climb.  After a couple of thousand requests (depending upon heap size), you'll start to encounter OOMEs.

Actual results:

NIO connector leaks memory

Expected results:

NIO connector doesn't leak memory


Additional info:

It looks like we should also backport the following from upstream tomcat 6 to address this:

------------------------------------------------------------------------
r1061524 | kkolinko | 2011-01-20 16:28:11 -0500 (Thu, 20 Jan 2011) | 2 lines

Additional patch (not important) for r1061421
Use the new release(s,p) method added in r1061421
------------------------------------------------------------------------
r1061421 | markt | 2011-01-20 12:55:51 -0500 (Thu, 20 Jan 2011) | 1 line

Fix very large memory leak in NIO connector
------------------------------------------------------------------------
r1033341 | kkolinko | 2010-11-09 22:27:42 -0500 (Tue, 09 Nov 2010) | 3 lines

Improve recycling of processors in Http11NioProtocol:
always call processor.recycle() before calling recycledProcessors.offer(processor);

------------------------------------------------------------------------

Comment 2 Chris Williams 2015-10-27 20:01:46 UTC
*** Bug 1272985 has been marked as a duplicate of this bug. ***

Comment 5 Coty Sutherland 2015-12-11 13:51:44 UTC
In addition to the revs Aaron provided, it also needs:
------------------------------------------------------------------------
r1064252 | kkolinko | 2011-01-27 13:43:18 -0500 (Thu, 27 Jan 2011) | 4 lines

Fix https://issues.apache.org/bugzilla/show_bug.cgi?id=50651
Fix NPE in InternalNioOutputBuffer.recycle().
- Do not throw NPE when recycle() is called more than once.
- Do not recycle the request in CoyoteAdapter#log() if it was not created there.
------------------------------------------------------------------------

Comment 18 errata-xmlrpc 2016-05-11 00:05:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0870.html


Note You need to log in before you can comment on or make changes to this bug.