Description of problem: Tomcat NIO connector has a very obvious memory leak introduced as a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1031327. The NIO connector does not properly offer processors back to the recycledProcessors queue. Thus, the processors are not cleaned up from the o.a.c.RequestGroupInfo's processors list, leading to an easy OOME as Tomcat keeps creating more and more processors as they are never recycled and indefinitely kept in the RequestGroupInfo processors list. It looks like NIO leaks such as this existed in some capacity before bz-1031327, but this made it even more evident by removing the most common recycledProccessors.offer call. Version-Release number of selected component (if applicable): 6.0.24-90 How reproducible: Very. Steps to Reproduce: 1. Configure your server.xml to use NIO: <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol" 2. Make repeated requests to the NIO connector. For instance, the following script to do so easily: #!/bin/bash N_REQ=0 TOTAL_REQUESTS=100000 while [ $((N_REQ)) -lt $((TOTAL_REQUESTS)) ] do curl "http://localhost:8080/sample/hello" > /dev/null 2>&1 N_REQ=$((N_REQ +1)) echo $N_REQ done 3. Note that heap usage continues to climb. After a couple of thousand requests (depending upon heap size), you'll start to encounter OOMEs. Actual results: NIO connector leaks memory Expected results: NIO connector doesn't leak memory Additional info: It looks like we should also backport the following from upstream tomcat 6 to address this: ------------------------------------------------------------------------ r1061524 | kkolinko | 2011-01-20 16:28:11 -0500 (Thu, 20 Jan 2011) | 2 lines Additional patch (not important) for r1061421 Use the new release(s,p) method added in r1061421 ------------------------------------------------------------------------ r1061421 | markt | 2011-01-20 12:55:51 -0500 (Thu, 20 Jan 2011) | 1 line Fix very large memory leak in NIO connector ------------------------------------------------------------------------ r1033341 | kkolinko | 2010-11-09 22:27:42 -0500 (Tue, 09 Nov 2010) | 3 lines Improve recycling of processors in Http11NioProtocol: always call processor.recycle() before calling recycledProcessors.offer(processor); ------------------------------------------------------------------------
*** Bug 1272985 has been marked as a duplicate of this bug. ***
In addition to the revs Aaron provided, it also needs: ------------------------------------------------------------------------ r1064252 | kkolinko | 2011-01-27 13:43:18 -0500 (Thu, 27 Jan 2011) | 4 lines Fix https://issues.apache.org/bugzilla/show_bug.cgi?id=50651 Fix NPE in InternalNioOutputBuffer.recycle(). - Do not throw NPE when recycle() is called more than once. - Do not recycle the request in CoyoteAdapter#log() if it was not created there. ------------------------------------------------------------------------
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0870.html