Hide Forgot
project_key: EDG recent 6hr soak test of hotrod server module showed some suspicious results: https://docspace.corp.redhat.com/docs/DOC-58120 Investigation is needed, because this is potential memory leak.
I'm in the process of creating another soak test run, where memory histogram will be taken with command {code} jmap -histo:live <pid> {code} every 30 minutes.
8hr check soak test with heap dump collection was performed, heap usage can be seen here: https://docspace.corp.redhat.com/docs/DOC-61819 heap dumps were taken with {code} jmap -dump:live,format=b,file=<file> <pid> {code} Heap dumps available here: http://dev39.qa.atl.jboss.com/~mlinhard/soak_tests/ File name structure: {code} jmap_dump_HHHHHH_DDDD_DD_DD_DD_DD_DD_CC_MMMM.zip jmap_dump_perf20_2011_04_07_12_22_23_00_0030.zip {code} H - host (host the dum comes from) D - date (system time when dump took place) C - counter (order of the dump taken for that particular JVM) M - minute (how many minutes after start was this taken)
I loaded the dump files (specifically jmap_dump_perf20_2011_04_07_19_45_17_03_0470.hprof) in Eclipse MAT and the biggest "dominator" is one instance of org.jgroups.protocols.pbcast.NAKACK that keeps a receiver window with 22870 instances of org.jgroups.Message for a total of ~25MB. The other dumps for perf20 have smaller numbers of messages, so I think this is the reason why the memory graph was trending upwards in the chart. I was able to find the logical name of the server: perf20-59609. It would seem that JGroups is not able to ack messages sent by itself and so accumulates a very large backlog in its NAK receiver window.
Dan, Galder do we have any conclusion on this ?
Michal, I think Dan's been looking into this, so I'm assigning it to him.
Link: Added: This issue relates to ISPN-1102
After investigating with Bela we realized that this was not a memory leak, the NAKACK receiver window was holding a lot of memory only because our buffers are 500 bytes and the useful information (the one counted by JGroups) is 50 bytes. Galder created issue ISPN-1102 to use smaller buffers.
Release Notes Docs Status: Added: Not Required
Release Notes Docs Status: Removed: Not Required Docs QE Status: Removed: ASSIGNED