Description of problem:

Tomcat NIO connector has a very obvious memory leak introduced as a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1031327.  The NIO connector does not properly offer processors back to the recycledProcessors queue.  Thus, the processors are not cleaned up from the o.a.c.RequestGroupInfo's processors list, leading to an easy OOME as Tomcat keeps creating more and more processors as they are never recycled and indefinitely kept in the RequestGroupInfo processors list.

It looks like NIO leaks such as this existed in some capacity before bz-1031327, but this made it even more evident by removing the most common recycledProccessors.offer call.


Version-Release number of selected component (if applicable):

6.0.24-90

How reproducible:

Very.


Steps to Reproduce:
1. Configure your server.xml to use NIO:

<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"

2. Make repeated requests to the NIO connector.  For instance, the following script to do so easily:

#!/bin/bash
N_REQ=0
TOTAL_REQUESTS=100000
while [ $((N_REQ)) -lt $((TOTAL_REQUESTS)) ]
do
curl "http://localhost:8080/sample/hello" > /dev/null 2>&1
N_REQ=$((N_REQ +1))
echo $N_REQ
done

3. Note that heap usage continues to climb.  After a couple of thousand requests (depending upon heap size), you'll start to encounter OOMEs.

Actual results:

NIO connector leaks memory

Expected results:

NIO connector doesn't leak memory


Additional info:

It looks like we should also backport the following from upstream tomcat 6 to address this:

------------------------------------------------------------------------
r1061524 | kkolinko | 2011-01-20 16:28:11 -0500 (Thu, 20 Jan 2011) | 2 lines

Additional patch (not important) for r1061421
Use the new release(s,p) method added in r1061421
------------------------------------------------------------------------
r1061421 | markt | 2011-01-20 12:55:51 -0500 (Thu, 20 Jan 2011) | 1 line

Fix very large memory leak in NIO connector
------------------------------------------------------------------------
r1033341 | kkolinko | 2010-11-09 22:27:42 -0500 (Tue, 09 Nov 2010) | 3 lines

Improve recycling of processors in Http11NioProtocol:
always call processor.recycle() before calling recycledProcessors.offer(processor);

------------------------------------------------------------------------