Bug 1273368 - building large project multiple times with incremented versionId would consume OldGen heap gradually
building large project multiple times with incremented versionId would consu...
Status: VERIFIED
Product: JBoss BRMS Platform 6
Classification: JBoss
Component: Business Central (Show other bugs)
6.1.0
All All
high Severity high
: ER5
: 6.2.0
Assigned To: Shelly McGowan
Jiri Locker
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-20 06:00 EDT by Hiroko Miura
Modified: 2016-09-20 01:13 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Enhancement
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Project1 (22456K) 100 releases (194.71 KB, text/plain)
2015-11-23 15:02 EST, Jiri Locker
no flags Details
Project2 (59946K) 30 releases (74.99 KB, text/plain)
2015-11-23 15:03 EST, Jiri Locker
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker JBPM-4807 Major Resolved building large project multiple times with incremented versionId would consume OldGen heap gradually 2016-11-08 07:08 EST
JBoss Issue Tracker PRODMGT-1419 Major Closed building large project multiple times with incremented versionId would consume OldGen heap gradually 2016-11-08 07:08 EST

  None (edit)
Description Hiroko Miura 2015-10-20 06:00:27 EDT
Description of problem:

Customer reported that OldGen of JVM heap of Business Central is increasing gradually when performing "Build & Deploy" with incremented fixed version again and again like the following.

2015-10-14T14:49:34.866+0900: 246.424: [Full GC [PSYoungGen: 576K->0K(690688K)] [ParOldGen: 450618K->223457K(1398272K)]
2015-10-14T14:53:09.703+0900: 461.260: [Full GC [PSYoungGen: 13192K->0K(669184K)] [ParOldGen: 615858K->365108K(1398272K)]
2015-10-14T14:57:18.456+0900: 710.013: [Full GC [PSYoungGen: 11991K->0K(670208K)] [ParOldGen: 671706K->435474K(1398272K)]
2015-10-14T14:58:45.999+0900: 797.556: [Full GC [PSYoungGen: 28149K->0K(659456K)] [ParOldGen: 556869K->447817K(1398272K)]
2015-10-14T15:01:51.246+0900: 982.804: [Full GC [PSYoungGen: 800K->0K(670720K)] [ParOldGen: 735296K->531696K(1398272K)]
2015-10-14T15:04:36.893+0900: 1148.450: [Full GC [PSYoungGen: 6986K->0K(671232K)] [ParOldGen: 771705K->602130K(1398272K)]
2015-10-14T15:06:05.781+0900: 1237.338: [Full GC [PSYoungGen: 8080K->0K(671232K)] [ParOldGen: 828260K->656776K(1398272K)]
2015-10-14T15:09:46.687+0900: 1458.245: [Full GC [PSYoungGen: 26619K->0K(671744K)] [ParOldGen: 691569K->641541K(1398272K)]

In this test, I build same project 40 times with incremented versionId.
This doesn't happen if specified version is identical fixed  version or SNAPSHOT version.

Whenever building module, KieModule which is built is stored
on memory KieRepository by the following method.

org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo.store()

Member variable 'KieModules' in class 

org.drools.compiler.kie.builder.impl.KieRepositoryImpl

keeps these modules per version like following.

kieModules        HashMap<K,V>  (id=1013)        
        [0]        HashMap$Entry<K,V>  (id=1031)        
        [1]        HashMap$Entry<K,V>  (id=1032)        
        [2]        HashMap$Entry<K,V>  (id=1092)        
        [3]        HashMap$Entry<K,V>  (id=1033)        
        [4]        HashMap$Entry<K,V>  (id=1034)        
                key        "testgroup:project1" (id=1037)        
                value        TreeMap<K,V>  (id=1038)        
                        [0]        TreeMap$Entry<K,V>  (id=1259)        
                                key        KieRepositoryImpl$ComparableVersion  (id=1265)        
                                        canonical        "(2,4)" (id=1268)        
                                        items        KieRepositoryImpl$ComparableVersion$ListItem  (id=1269)        
                                        value        "2.4.0" (id=1270)        
                                value        MemoryKieModule  (id=1266)        
                        [1]        TreeMap$Entry<K,V>  (id=1261)        
                                key        KieRepositoryImpl$ComparableVersion  (id=1272)        
                                        canonical        "(2,5)" (id=1275)        
                                        items        KieRepositoryImpl$ComparableVersion$ListItem  (id=1277)        
                                        value        "2.5.0" (id=1279)        
                                value        MemoryKieModule  (id=1123)        
                        [2]        TreeMap$Entry<K,V>  (id=1263)        
                                key        KieRepositoryImpl$ComparableVersion  (id=1226)        
                                        canonical        "(2,5,1)" (id=1283)        
                                        items        KieRepositoryImpl$ComparableVersion$ListItem  (id=1285)        
                                        value        "2.5.1" (id=1287)        
                                value        MemoryKieModule  (id=1217)        

In above example, this project is built 3 times with 3 version(2.4.0, 2.5.0 and 2.5.1) respectively. 
Therefore, this KieModules could be getting large if the project gets built with incremented fixed version again and again, especially in case that  kieModule is large (in this customer's case, kjar size is more than 4MB).

Version-Release number of selected component (if applicable):
6.1.x


How reproducible:
Build large project again and again with different versionId.

Steps to Reproduce:
0. prepare large project (which has large and many XLS decision tables)
1. set versionId 1.0 and then build it
2. set versionId 1.1 and then build it
3. set versionID 1.2 and then build it
    :

Actual results:
OldGern heap is consumed gradually 


Expected results:
OldGen heap is not consumed so much 

Additional info:
Comment 1 Mario Fusco 2015-10-20 08:11:15 EDT
Marco suggested to store the older versions in a LRU cache. Since he already developed a LRU cache (for a different purpose) in another PR related with the KieRepo I'm reassigning this ticket to him.
Comment 2 Mario Fusco 2015-10-21 03:42:26 EDT
Now the kmodules are stored in a 2 levels LRU cache. This cache is configurable with the following 2 system properties:

kie.repository.project.cache.size (default value: 100) -> number of different projects retained by the cache

kie.repository.project.versions.cache.size (default value: 10) -> number of different versions for each project retained by the cache  

Fixed by https://github.com/droolsjbpm/drools/commit/b44ab2f85
Comment 3 Alessandro Lazarotti 2015-10-22 14:17:01 EDT
I think appropriated document this feature to allow users tune their environments based in this new property. Could it be documented please ?
Comment 5 Jiri Locker 2015-11-23 14:57:49 EST
I have verified that kmodules cache is now limited and its size is configurable.

I tested with two large projects. In both cases I first reproduced heap size OOME. Then lowered cache parameters and then built approx. 3 times as many releases of the project than the number of releases that caused OOME.

Project 1
- size: 22456K
- crashed in release: 33 (default cache parameters)
- successfully performed 100 releases with:
  - kie.repository.project.cache.size=2
  - kie.repository.project.versions.cache.size=10

Project 2
- size: 59946K
- crashed in release: 12 (default cache parameters)
- successfully performed 30 releases with:
  - kie.repository.project.cache.size=2
  - kie.repository.project.versions.cache.size=3

Maximum heap size in all cases was 1303m. Following attachments are GC logs of the successful releases with limited cache. ParOldGen is almost full but its size is stable. PSYoungGen is regularly cleaned to 0K.
Comment 6 Jiri Locker 2015-11-23 15:02 EST
Created attachment 1097836 [details]
Project1 (22456K) 100 releases

kie.repository.project.cache.size=2
kie.repository.project.versions.cache.size=10
Comment 7 Jiri Locker 2015-11-23 15:03 EST
Created attachment 1097837 [details]
Project2 (59946K) 30 releases

kie.repository.project.cache.size=2
kie.repository.project.versions.cache.size=3
Comment 8 Jiri Locker 2015-11-23 15:15:53 EST
Important note: cache size only limits number of released KIE modules (KJARs). It doesn't really limit memory consumption. If the cache is full and a new release is made, number of cached KIE modules will not grow but the occupied memory may grow if the newly released KJAR size is greater than size of the KJAR that was evicted.

It may be a good idea to only cache a single released KIE module in mission-critical system by using:

kie.repository.project.cache.size=1
kie.repository.project.versions.cache.size=1

or to disable the cache completely. This depends on whether there are some other negative effects (performance-wise) of using very small cache or not using it at all. This should be covered in documentation.
Comment 9 JBoss JIRA Server 2016-08-24 10:53:21 EDT
Marco Rietveld <marco.rietveld@redhat.com> updated the status of jira JBPM-4807 to Resolved

Note You need to log in before you can comment on or make changes to this bug.