Description of problem:
HotSpot leaking memory in long-running requests. While checking the RSS part, it is seen memory usage is going high gradually with time.
# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7.3
Steps to Reproduce:
1. Check out the sample project from GitHub onto a RHEL 7 system.
2. cd into nacroleptic
3. Run # mvn clean package
4. Run java -jar target/narcoleptic.jar
5. cd into ../devourer
6. Run # mvn clean package
7. Run ./run.sh
8. Download Apache JMeter 3.0 from http://jmeter.apache.org/download_jmeter.cgi
Extract the archive and start with the "bin/jmeter.sh" script.
9. Open the file "load_test.jmx" with JMeter.
10. Start the JMeter tests with CTRL+R
Leave for *at least* 30 minutes, then come back and observe the amount of memory used by the process. It will have increased beyond what would be expected. Leave for several hours, maybe a couple of days. Memory usage will continue to increase.
Checked from RSS values under ps/top.
Tried with running the application under valgrind --tool=massif, but the application gets killed after a few seconds.
Memory usage of the application increases over time, without stopping. It may slow, and *appear* to stop, but given a large enough time span, it will *always* continue to increase.
Memory usage of the application remains stable.
Upstream bugs :
Fix appears to be added to 8u152 and is backported to 8u131 b31. We need the fix backported to RHEL OpenJDK or an openjdk update.
With this already being backported to 8u131 b31, I suspect it will form part of 8u141 next month.
(In reply to Andrew John Hughes from comment #2)
> With this already being backported to 8u131 b31, I suspect it will form part
> of 8u141 next month.
I see this is moved to RHEL 7.5
Does this mean this will not be fixed in the openjdk update coming with rhel-7.4?
(In reply to Deepu K S from comment #3)
> (In reply to Andrew John Hughes from comment #2)
> > With this already being backported to 8u131 b31, I suspect it will form part
> > of 8u141 next month.
> I see this is moved to RHEL 7.5
> Does this mean this will not be fixed in the openjdk update coming with
Yes, it was submitted too late to make rhel-7.4.
It also doesn't appear to be included in 8u141 as we've received it.
The fastest route to get a fix in 7.4 will be to request z-stream once this bug is resolved (waiting on PM approval; I'll ping them today)
This is fixed in the upcoming 7.5. If it's still desired in 7.4, then now is the time to request it for z-stream.
According to https://bugs.openjdk.java.net/browse/JDK-8164293.
Isn't the build for u144 and earlier is b32?
(In reply to Ding-Yi Chen from comment #8)
> According to https://bugs.openjdk.java.net/browse/JDK-8164293.
> Isn't the build for u144 and earlier is b32?
It's not present in 8u144.
Those b31/b32 fixes are misleading. They seem to be builds made for specific Oracle customers only, as far as I can gather.
The relevant bits are the fixed version (8u152) and the line for 8u161 b01.
8u161 is the January 2018 security update, which will be based on the 8u152 feature release expected next week.
So, this should be fixed everywhere by the time of the January CPU. I expect us to try and get 8u152 into RHEL 7 some time in November.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.