Bug 1156297
| Summary: | Classloader leak by threadLocal | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Retired] JBoss BPMS Platform 6 | Reporter: | Toshiya Kobayashi <tkobayas> | ||||
| Component: | BRE | Assignee: | Mario Fusco <mfusco> | ||||
| Status: | CLOSED EOL | QA Contact: | Marek Winkler <mwinkler> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 6.0.2 | CC: | alazarot, bpms-support, etirelli, ksuzumur, kverlaen, mbaluch, mfusco, mwinkler, rrajasek, tkobayas | ||||
| Target Milestone: | ER5 | ||||||
| Target Release: | 6.1.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-03-27 20:09:54 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 950241 [details]
rewards-basic-threadLocalLeak.zip
I fixed the problem related with UpgradableReentrantReadWriteLock with this commit https://github.com/droolsjbpm/drools/commit/58d8025cd The other leaks are caused by mvel and I need a bit more time to investigate/fix them. Very likely a new mvel release will be necessary too. I fixed it in the new version of mvel 2.2.3.Final + this https://github.com/droolsjbpm/drools/commit/2d86a327fae6fd0496ad5241191e23173a892f1a I have verified that the fix is included into BRMS 6.1.0 ER6 (6.2.0.Final-redhat-2). I have followed the steps to reproduce and I can confirm that although I still see 7 "deployment.rewards-basic-6.0.2-SNAPSHOT.war" rows, when I right click each row -> "Path to GC Root" -> "exclude all phantom/weak/soft etc. references", the result set is empty for each "deployment.rewards-basic-6.0.2-SNAPSHOT.war" row. (*) When I tested the same scenario with 6.2.0.CR4 version, the result set contained rows referring to org.mvel2.ParserContext, therefore I assume the original problem has been fixed. Toshiya, could you please confirm that the difference from your described expected behaviour is acceptable? Meanwhile, I am marking this BZ as verified, we can reopen it if needed. Many thanks! (*) After the fix, Eclipse Memory Analyzer reports the following Leak suspects (which I consider not to be caused by BRMS/BPMS): Problem suspect 1 The class "java.lang.ref.Finalizer", loaded by "<system class loader>", occupies 42,346,120 (19.57%) bytes. Keywords java.lang.ref.Finalizer Problem suspect 2 210 instances of "org.jboss.modules.ModuleClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0xb68a06e0" occupy 132,444,632 (61.21%) bytes. Biggest instances: org.jboss.modules.ModuleClassLoader @ 0xc01f40e0 - 19,882,512 (9.19%) bytes. org.jboss.modules.ModuleClassLoader @ 0xbdd1fbb0 - 19,881,680 (9.19%) bytes. org.jboss.modules.ModuleClassLoader @ 0xc23dc3a0 - 19,809,144 (9.16%) bytes. org.jboss.modules.ModuleClassLoader @ 0xb97b5290 - 19,798,280 (9.15%) bytes. org.jboss.modules.ModuleClassLoader @ 0xbb85f670 - 19,798,000 (9.15%) bytes. org.jboss.modules.ModuleClassLoader @ 0xb78f68d8 - 19,774,840 (9.14%) bytes. org.jboss.modules.ModuleClassLoader @ 0xc481bd50 - 9,735,960 (4.50%) bytes. Keywords sun.misc.Launcher$AppClassLoader @ 0xb68a06e0 org.jboss.modules.ModuleClassLoader |
Description of problem: When you repeat deploy/undeploy an web application in JBoss EAP, which embeds jBPM libraries, it will finally results in perm gen OutOfMemoryError. Analyzing a heap dump by Eclipse Memory Analyzer, I found 3 kinds of ModuleClassLoader retention (= fails to be garbage-collected on undeployment). A) Class Name | Shallow Heap | Retained Heap --------------------------------------------------------------------------------------------------------------------------------------------- org.jboss.modules.ModuleClassLoader @ 0xc67d2978 | 160 | 13,014,408 |- <classloader> class org.drools.core.common.UpgradableReentrantReadWriteLock$LockRequestCounter @ 0xc85ff5c8| 0 | 0 | '- <class> org.drools.core.common.UpgradableReentrantReadWriteLock$LockRequestCounter @ 0xc8a77aa0 | 24 | 24 | '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xc8a77a80 | 56 | 80 | '- [164] java.lang.ThreadLocal$ThreadLocalMap$Entry[256] @ 0xd20b3ea8 | 2,072 | 33,008 | '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xb6d8a9a0 | 32 | 33,040 | '- threadLocals java.lang.Thread @ 0xbb1403a0 http-/127.0.0.1:8080-1 Thread | 160 | 33,504 --------------------------------------------------------------------------------------------------------------------------------------------- B) Class Name | Shallow Heap | Retained Heap -------------------------------------------------------------------------------------------------------------------------- org.jboss.modules.ModuleClassLoader @ 0xc407ad50 | 160 | 13,007,840 |- <classloader> class org.mvel2.debug.DebuggerContext @ 0xc57363c0 | 0 | 0 | '- <class> org.mvel2.debug.DebuggerContext @ 0xc58c7370 | 40 | 128 | '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xc58c7350 | 56 | 184 | '- [183] java.lang.ThreadLocal$ThreadLocalMap$Entry[256] @ 0xd20b3ea8 | 2,072 | 33,008 | '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xb6d8a9a0 | 32 | 33,040 | '- threadLocals java.lang.Thread @ 0xbb1403a0 http-/127.0.0.1:8080-1 Thread| 160 | 33,504 -------------------------------------------------------------------------------------------------------------------------- C) Class Name | Shallow Heap | Retained Heap ---------------------------------------------------------------------------------------------------------------------------------------- org.jboss.modules.ModuleClassLoader @ 0xbe2fd730 | 160 | 24,288,552 '- <classloader> class org.mvel2.ParserContext @ 0xbf02ee08 | 0 | 0 '- <class> org.mvel2.ParserContext @ 0xbf0548b8 | 168 | 1,376 '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xbf054898 | 56 | 1,432 '- [0] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xba82deb8 | 1,048 | 35,168 '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xb8d149b8 | 32 | 35,200 '- threadLocals java.lang.Thread @ 0xb8d14868 http-/127.0.0.1:8080-1 Native Stack, Thread| 160 | 35,736 ---------------------------------------------------------------------------------------------------------------------------------------- Steps to Reproduce: 1. Unzip attached rewards-basic-threadLocalLeak.zip 2. mvn clean install 3. deploy target/rewards-basic-6.0.2-SNAPSHOT.war to JBoss EAP 4. Access to http://localhost:8080/rewards-basic 5. Click "Start Reward Process" 6. Press "Start a process" 7. Redeploy the application (e.g. $ mv rewards-basic-6.0.2-SNAPSHOT.war.deployed rewards-basic-6.0.2-SNAPSHOT.war.dodeploy) 8. repeat 5-7 (5 times) 9. Capture a heap dump (e.g. jmap -dump:format=b,file=heap.bin <JAVA_PID>) 10. Open the heap dump with Eclipse Memory Analyzer 11. Run "SELECT module.identifier.name.value.toString() FROM org.jboss.modules.ModuleClassLoader" with OQL tab. 12. You will find several "deployment.rewards-basic-6.0.2-SNAPSHOT.war" rows 13. Right click each row -> "Path to GC Root" -> "exclude all phontom/weak/soft etc. references" Actual results: You will see that the ModuleClassLoader is retained by one of UpgradableReentrantReadWriteLock$LockRequestCounter, DebuggerContext, ParserContext. Expected results: You don't see several "deployment.rewards-basic-6.0.2-SNAPSHOT.war" rows. (I think 2 rows are acceptable. One is alive one, the other is retained by org.h2.engine.DatabaseCloser)