Description of problem: Customer reported that OldGen of JVM heap of Business Central is increasing gradually when performing "Build & Deploy" with incremented fixed version again and again like the following. 2015-10-14T14:49:34.866+0900: 246.424: [Full GC [PSYoungGen: 576K->0K(690688K)] [ParOldGen: 450618K->223457K(1398272K)] 2015-10-14T14:53:09.703+0900: 461.260: [Full GC [PSYoungGen: 13192K->0K(669184K)] [ParOldGen: 615858K->365108K(1398272K)] 2015-10-14T14:57:18.456+0900: 710.013: [Full GC [PSYoungGen: 11991K->0K(670208K)] [ParOldGen: 671706K->435474K(1398272K)] 2015-10-14T14:58:45.999+0900: 797.556: [Full GC [PSYoungGen: 28149K->0K(659456K)] [ParOldGen: 556869K->447817K(1398272K)] 2015-10-14T15:01:51.246+0900: 982.804: [Full GC [PSYoungGen: 800K->0K(670720K)] [ParOldGen: 735296K->531696K(1398272K)] 2015-10-14T15:04:36.893+0900: 1148.450: [Full GC [PSYoungGen: 6986K->0K(671232K)] [ParOldGen: 771705K->602130K(1398272K)] 2015-10-14T15:06:05.781+0900: 1237.338: [Full GC [PSYoungGen: 8080K->0K(671232K)] [ParOldGen: 828260K->656776K(1398272K)] 2015-10-14T15:09:46.687+0900: 1458.245: [Full GC [PSYoungGen: 26619K->0K(671744K)] [ParOldGen: 691569K->641541K(1398272K)] In this test, I build same project 40 times with incremented versionId. This doesn't happen if specified version is identical fixed version or SNAPSHOT version. Whenever building module, KieModule which is built is stored on memory KieRepository by the following method. org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo.store() Member variable 'KieModules' in class org.drools.compiler.kie.builder.impl.KieRepositoryImpl keeps these modules per version like following. kieModules HashMap<K,V> (id=1013) [0] HashMap$Entry<K,V> (id=1031) [1] HashMap$Entry<K,V> (id=1032) [2] HashMap$Entry<K,V> (id=1092) [3] HashMap$Entry<K,V> (id=1033) [4] HashMap$Entry<K,V> (id=1034) key "testgroup:project1" (id=1037) value TreeMap<K,V> (id=1038) [0] TreeMap$Entry<K,V> (id=1259) key KieRepositoryImpl$ComparableVersion (id=1265) canonical "(2,4)" (id=1268) items KieRepositoryImpl$ComparableVersion$ListItem (id=1269) value "2.4.0" (id=1270) value MemoryKieModule (id=1266) [1] TreeMap$Entry<K,V> (id=1261) key KieRepositoryImpl$ComparableVersion (id=1272) canonical "(2,5)" (id=1275) items KieRepositoryImpl$ComparableVersion$ListItem (id=1277) value "2.5.0" (id=1279) value MemoryKieModule (id=1123) [2] TreeMap$Entry<K,V> (id=1263) key KieRepositoryImpl$ComparableVersion (id=1226) canonical "(2,5,1)" (id=1283) items KieRepositoryImpl$ComparableVersion$ListItem (id=1285) value "2.5.1" (id=1287) value MemoryKieModule (id=1217) In above example, this project is built 3 times with 3 version(2.4.0, 2.5.0 and 2.5.1) respectively. Therefore, this KieModules could be getting large if the project gets built with incremented fixed version again and again, especially in case that kieModule is large (in this customer's case, kjar size is more than 4MB). Version-Release number of selected component (if applicable): 6.1.x How reproducible: Build large project again and again with different versionId. Steps to Reproduce: 0. prepare large project (which has large and many XLS decision tables) 1. set versionId 1.0 and then build it 2. set versionId 1.1 and then build it 3. set versionID 1.2 and then build it : Actual results: OldGern heap is consumed gradually Expected results: OldGen heap is not consumed so much Additional info:
Marco suggested to store the older versions in a LRU cache. Since he already developed a LRU cache (for a different purpose) in another PR related with the KieRepo I'm reassigning this ticket to him.
Now the kmodules are stored in a 2 levels LRU cache. This cache is configurable with the following 2 system properties: kie.repository.project.cache.size (default value: 100) -> number of different projects retained by the cache kie.repository.project.versions.cache.size (default value: 10) -> number of different versions for each project retained by the cache Fixed by https://github.com/droolsjbpm/drools/commit/b44ab2f85
I think appropriated document this feature to allow users tune their environments based in this new property. Could it be documented please ?
I have verified that kmodules cache is now limited and its size is configurable. I tested with two large projects. In both cases I first reproduced heap size OOME. Then lowered cache parameters and then built approx. 3 times as many releases of the project than the number of releases that caused OOME. Project 1 - size: 22456K - crashed in release: 33 (default cache parameters) - successfully performed 100 releases with: - kie.repository.project.cache.size=2 - kie.repository.project.versions.cache.size=10 Project 2 - size: 59946K - crashed in release: 12 (default cache parameters) - successfully performed 30 releases with: - kie.repository.project.cache.size=2 - kie.repository.project.versions.cache.size=3 Maximum heap size in all cases was 1303m. Following attachments are GC logs of the successful releases with limited cache. ParOldGen is almost full but its size is stable. PSYoungGen is regularly cleaned to 0K.
Created attachment 1097836 [details] Project1 (22456K) 100 releases kie.repository.project.cache.size=2 kie.repository.project.versions.cache.size=10
Created attachment 1097837 [details] Project2 (59946K) 30 releases kie.repository.project.cache.size=2 kie.repository.project.versions.cache.size=3
Important note: cache size only limits number of released KIE modules (KJARs). It doesn't really limit memory consumption. If the cache is full and a new release is made, number of cached KIE modules will not grow but the occupied memory may grow if the newly released KJAR size is greater than size of the KJAR that was evicted. It may be a good idea to only cache a single released KIE module in mission-critical system by using: kie.repository.project.cache.size=1 kie.repository.project.versions.cache.size=1 or to disable the cache completely. This depends on whether there are some other negative effects (performance-wise) of using very small cache or not using it at all. This should be covered in documentation.
Marco Rietveld <marco.rietveld> updated the status of jira JBPM-4807 to Resolved