Bug 1003600

Summary: Memory leak in CDI / JPA integration code
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Tomaz Cerar <tcerar>
Component: CDI/Weld, JPAAssignee: Tomaz Cerar <tcerar>
Status: CLOSED CURRENTRELEASE QA Contact: Marek Schmidt <maschmid>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.1.0, 6.2.0, 6.1.1CC: brian.stansberry, myarboro, pmuir, sdouglas
Target Milestone: ER6   
Target Release: EAP 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
URL: https://community.jboss.org/thread/231877
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-15 16:21:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1010467    
Bug Blocks: 1033754    

Description Tomaz Cerar 2013-09-02 13:10:40 UTC
WeldJpaInjectionServices holds hard reference to DeploymentUnit which causes DeploymentUnit and all everything resulting from it to not get cleaned up.

Same problems was fixed in upstream for WildFly 8 alpha3 as part of 
https://github.com/wildfly/wildfly/pull/4795

see for some more details.
http://lists.jboss.org/pipermail/wildfly-dev/2013-July/000466.html

Comment 2 Stuart Douglas 2013-09-02 15:21:42 UTC
I don't see how this is possible. WeldJpaInjectionServices is only held by the weld bootstrap service, which should be removed once the deployment is removed.

Comment 3 Tomaz Cerar 2013-09-02 16:21:23 UTC
I misspoke, problem is ServiceRegistry reference, which is never cleaned up during undeploy as it is server-wide resource.

Comment 4 Stuart Douglas 2013-09-02 17:18:20 UTC
I'm still not sure how the ServiceRegistry reference can be a problem. The WeldJpaInjectionServices instance will not be referenced, so it should still be GC'ed.

Comment 5 Tomaz Cerar 2013-09-03 21:35:56 UTC
Stuart, this is example of leak taken from yourkit
Paths from "GC Roots" to "Object DeploymentUnitImpl #597743"

+-------------------------------------------------------------------------------------------------+-----------------+----------------+
|                                              Name                                               |  Retained Size  |  Shallow Size  |
+-------------------------------------------------------------------------------------------------+-----------------+----------------+
|  +---org.jboss.as.server.deployment.DeploymentUnitImpl                                          |        916.016  |            24  |
|    |                                                                                            |                 |                |
|    +---deploymentUnit of org.jboss.as.weld.services.bootstrap.WeldJpaInjectionServices          |        916.032  |            16  |
|      |                                                                                          |                 |                |
|      +---value of java.util.HashMap$Entry                                                       |             48  |            24  |
|        |                                                                                        |                 |                |
|        +---[27] of java.util.HashMap$Entry[32]                                                  |            944  |           144  |
|          |                                                                                      |                 |                |
|          +---table of java.util.HashMap                                                         |          1.008  |            48  |
|            |                                                                                    |                 |                |
|            +---services of org.jboss.weld.bootstrap.api.helpers.SimpleServiceRegistry           |          1.024  |            16  |
|              |                                                                                  |                 |                |
|              +---services of org.jboss.weld.manager.BeanManagerImpl                             |      1.235.320  |           120  |
|                |                                                                                |                 |                |
|                +---beanManager of org.jboss.arquillian.testenricher.cdi.container.CDIExtension  |      1.235.656  |           336  |
|                  |                                                                              |                 |                |
|                  +---[0] of java.lang.Object[320]                                               |      1.323.720  |         1.296  |
|                    |                                                                            |                 |                |
|                    +---elementData of java.util.Vector                                          |      1.323.744  |            24  |
|                      |                                                                          |                 |                |
|                      +---classes of org.jboss.modules.ModuleClassLoader                         |      1.588.192  |            80  |
|                        |                                                                        |                 |                |
|                        +---referent of java.util.WeakHashMap$Entry                              |            888  |            40  |
|                          |                                                                      |                 |                |
|                          +---[108] of java.util.WeakHashMap$Entry[128]                          |         25.968  |           528  |
|                            |                                                                    |                 |                |
|                            +---table of java.util.WeakHashMap                                   |         26.056  |            56  |
|                              |                                                                  |                 |                |
|                              +---loaderToCache of java.lang.reflect.Proxy [Class]               |         38.048  |           344  |
+-------------------------------------------------------------------------------------------------+-----------------+----------------+

Generated by YourKit Java Profiler 12.0.3 3.9.2013 23:34:48

Comment 6 Tomaz Cerar 2013-09-03 21:39:21 UTC
better formatting: http://pastebin.test.redhat.com/161784

this is from current 6.1.x branch aka same as 6.1.1.GA
problem is that DeploymentUnit instances are saying around after undeploy
with proposed fix, there are no deployment units left after undeploy.

Comment 7 Stuart Douglas 2013-09-04 06:36:34 UTC
No, that is an Arquillian leak. The BM is being held from org.jboss.arquillian.testenricher.cdi.container.CDIExtension#beanManager.

When reading these graphs you have to read them from the bottom up, not the top down, the closest thing to the bottom that should have been GC'ed is generally the culprit. 

I'm thinking this should be closed, as it does not affect EAP itself?

Comment 8 Tomaz Cerar 2013-09-04 10:27:40 UTC
I had same impression that the problem is with static reference to BeanManager in CDIExtension
see:
https://github.com/arquillian/arquillian-core/blob/master/testenrichers/cdi/src/main/java/org/jboss/arquillian/testenricher/cdi/container/CDIExtension.java

But given that CDI extensions are deployment scoped it should be GC-ed on undeploy.

Comment 9 Stuart Douglas 2013-09-04 10:37:29 UTC
No, instances of the extension will be GC'ed, however because the bean manager reference is held in a static field it will only be cleared when the class itself is GCed. 

Assuming that Arquillian has actually been undeployed we should be looking at what is still holding onto the class loader references that prevents the Arquillian deployments class loader from being GCed.

Comment 10 Tomaz Cerar 2013-09-04 13:39:01 UTC
Issue is related also with https://hibernate.atlassian.net/browse/HHH-8363

Comment 11 Tomaz Cerar 2013-09-04 13:41:22 UTC
After discussion with Stuart we both agree that is memory leak, but my proposed fix just mask the real memory leaks.

Currently we identified possible causes in not clearing TCCL in executor service and linked hibernate issue.

Comment 14 Tomaz Cerar 2013-09-17 22:08:44 UTC
This is not complete yet, linked PR was just first fix.
for complete fix hibernate upgrade is required.
Hibernate fix was already applied to upstream and is included in 4.3.0.Beta4
for EAP it needs 4.2.x release.
We can track that separately or as part of this issue.

Comment 20 Tomaz Cerar 2013-10-07 15:06:50 UTC
I can reproduce this, it is related to buggy h2 we use for ExampleDB datasource.
I will send PR to address this same way we did in upstream

Comment 21 Tomaz Cerar 2013-10-07 15:54:16 UTC
https://github.com/jbossas/jboss-eap/pull/515
This PR changes default ExampleDS configuration to disable H2 DatabaseCloser thread creation which can hold reference to hibernate CL.

This same problem exists also in this quickstart
https://github.com/wildfly/quickstart/blob/master/kitchensink-html5-mobile/src/test/resources/arquillian-ds.xml also needs this change 
;DB_CLOSE_ON_EXIT=FALSE should be added to url.

with that change memory leak does not appear anymore.

This problem is related only to h2 in memory database and does not affect any other jdbc drivers I was testing.

If I run quickstart with modified datasource this memory leak does not appear anymore

Comment 25 Marek Schmidt 2013-10-18 15:23:37 UTC
Verified on EAP 6.2.0.ER6. 

I have noticed the instances are still laying around in memory for quite a while after the deployment ends awaiting finalization, but I expect this to be just Java being lazy about finalization, as there are no GC roots to be found for them now...