Bug 1003987

Summary:	When doing remote EJB calls, the marshalling code creates a very large number of sun.reflect.GeneratedSerializedConstructor objects, taking the vast majority of the heap space, causing memory consumption issues.
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Andrig T Miller <anmiller>
Component:	Remoting	Assignee:	David M. Lloyd <david.lloyd>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jitka Kozana <jkudrnac>
Severity:	urgent	Docs Contact:	Russell Dickenson <rdickens>
Priority:	unspecified
Version:	6.2.0	CC:	bmaxwell, jdoyle, jmartisk, lthon, myarboro
Target Milestone:	ER1
Target Release:	EAP 6.2.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-12-15 16:20:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1003990, 1010377, 1012664, 1120777

Description Andrig T Miller 2013-09-03 16:26:31 UTC

Description of problem:

During our profiling of SPECjEnterprise2010, we discovered a very large number of sun.reflect.GeneratedSerializedConstructor* objects being creating, taking the vast majority of the heap space, and they are end up in the old generation, causing large garbage collection spikes.  This causes scalability problems with EAP.

Version-Release number of selected component (if applicable):

JBoss Marshalling 1.3.x


How reproducible:

Always, with any EJB remote calls.

Steps to Reproduce:
1.Run SPECjEnterprise2010 at tx1400 rate.
2.
3.

Actual results:

Over 2.5 GB's of memory in the old generation will be seen (JProfiler) just for two of these constructor objects, and more than 3GB's of memory consumption when you count all constructor object references.


Expected results:

Should be very few instances, with a very small memory footprint.

Additional info:

The 1.4.0 version that David Lloyd has tagged as Final today, contains the fix for this problem.

The EJB client library will have to be rebuilt along with JBoss Marshalling in order for this to work properly.

Comment 1 Andrig T Miller 2013-09-03 16:30:30 UTC

Also, I forgot to add that this issue is in JIRA as:

https://issues.jboss.org/browse/JBMAR-153

Comment 2 Andrig T Miller 2013-09-03 16:35:36 UTC

Changed the component to remoting, since JBoss Marshalling is a part of Remoting.

Comment 3 David M. Lloyd 2013-09-03 17:00:38 UTC

Initialized flags.

Comment 4 David M. Lloyd 2013-09-03 19:21:32 UTC

Please be sure to also ack https://bugzilla.redhat.com/show_bug.cgi?id=1003990 if you ack this issue.

Comment 7 Ladislav Thon 2013-09-24 10:00:43 UTC

Verified with EAP 6.2.0.ER1.

Comment 8 JBoss JIRA Server 2013-09-26 18:41:24 UTC

Brad Maxwell <bmaxwell> made a comment on jira JBMAR-153

Background info: Originally JBoss Marshalling used a weak map so that it would not leak memory when a deployment was undeployed.  This had a side effect, if a weak reference was cleared, JBoss Marshalling would generate a new constructor, except the JDK still kept the reference to the old one. So they would pile up over time, especially if you GC a lot in between invocations for whatever reason. The fix is to never regenerate them, by replacing the weak map with a strong map.  Since a strong map is used the API had to be changed so that on undeploy the map could be released and avoid a memory leak, which the JBoss EAP code calls when an application is undeployed.