Bug 1039585 - [GSS] (6.3.0) Clustered session memory leaking
Summary: [GSS] (6.3.0) Clustered session memory leaking
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: DR0
: EAP 6.3.0
Assignee: Paul Ferraro
QA Contact: Jitka Kozana
Russell Dickenson
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1030681 1051591 1130600
TreeView+ depends on / blocked
 
Reported: 2013-12-09 14:38 UTC by Shay Matasaro
Modified: 2018-12-06 15:33 UTC (History)
11 users (show)

(edit)
Previous versions of JBoss EAP 6 contained a bug that could lead to an OutOfMemoryException in distributed web sessions. The exception was encountered if a web session expired without the lock objects created by the session manager being released or destroyed. As web sessions continued to expire, the residual lock objects accumulated in memory.  Eventually, this would lead to an OutOfMemoryException.  The only recourse was to redeploy the web application. 

In this release of the product the lock objects are properly released and the OutOfMemory no longer presents
Clone Of:
(edit)
Last Closed: 2014-06-28 15:40:37 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker WFLY-406 Major Closed Redesign web session clustering 2018-06-22 07:45 UTC
Red Hat Bugzilla 1030681 None None None Never
JBoss Issue Tracker WFLY-898 Major Closed SharedLocalYieldingLockManager.LocalLock 2018-06-22 07:45 UTC

Internal Trackers: 1051591

Description Shay Matasaro 2013-12-09 14:38:55 UTC
Stale sessions are not getting removed from memory.
SharedLocalYieldingLockManager.LocalLock   hogs an increasing amount of heap space.

original case data https://bugzilla.redhat.com/show_bug.cgi?id=1030681

Possible source is https://issues.jboss.org/browse/WFLY-898, which was closed when https://issues.jboss.org/browse/WFLY-406  was submitted

Looking to port a fix for EAP 6 , but the fix for WFLY-406 is a complete rewrite and involves 321 files.

Is there a smaller scope fix that could be ported or a possible workaround?

Comment 1 Paul Ferraro 2013-12-09 15:09:00 UTC
https://community.jboss.org/thread/235116

Might this just be a product of the number of sessions in your application?
The SharedLocalYieldingClusterLockManager hold a LocalLock for every session "owned" by the local node.
However, if a session passivates, the local lock for that session stays in memory.  Thus is it possible to exhaust the heap via the number of session even when using passivation.  It might be a good idea to release/destroy the local lock on passivation too.
 
The LocalLock is meant to be destroyed when a session invalidates, expires, or when some other node takes the lock.  While it's certainly possible that one of these cases is not correctly destroying the LocalLock - I want to rule out the above cause first.

Comment 2 dereed 2013-12-31 16:15:18 UTC
This is definitely a bug in EAP.  It can be easily reproduced, and the leak occurs every time a session times out.  (It does not occur if session.invalidate is called).

Comment 3 Paul Ferraro 2014-01-03 21:48:10 UTC
https://github.com/jbossas/jboss-eap/pull/749

Comment 4 Shay Matasaro 2014-01-10 16:03:35 UTC
fix checked in for 6.2.0

Comment 5 Richard Janík 2014-02-19 15:21:00 UTC
Verified for 6.3.0.DR0.

Comment 6 Scott Dodson 2014-03-24 14:41:40 UTC
I'm working with the Linux performance teams on their efforts to do proactive performance analysis based on collection of sosreports. One of the things they hi-lighted is that our nodes are using a large amount of memory in Slabcache, in particular negative dentry cache entries.

We wrote a systemtap script and we've found that JBOSS is responsible for all of the negative dentry items, could this be related to this bug as well? What we see is JBOSS attempting to look for thousands of files that do not exist. My thinking is that perhaps it's looking for files associated with expired sessions 

[root@gss-webjava01 ~]# stap dentry.stp
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1675781120
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/1676442624
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1159221248
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/826576896
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-700282880
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-367345664
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-670302208
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/762603520
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/1131705344
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/546164736
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/348350464
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/1701886976
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1409987584
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1221157888
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1984292864
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1322153984
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-392439808
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1744696320
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/1643878400
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/249819136
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1412597760
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1280489472
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1320242176
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/-1055926272
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/1560241152
var/lib/jbossas/standalone/data/infinispan/web/default-host/ROOT/31753216

Comment 7 dereed 2014-03-24 14:54:10 UTC
> My thinking is that perhaps it's looking for files associated with expired sessions

That is the directory where the sessions are passivated.
So expired session IDs is the most likely cause.

That wouldn't be related to this BZ though.
That would just be the normal expected behavior for an expired session ID.

Comment 8 Scott Mumford 2014-05-05 05:02:05 UTC
Paul, can you provide some details in the Doc Text field as to what was causing this leak, and how you fixed it, so we can get it into the 6.3.0 Release Notes.

Cheers.

Comment 9 Paul Ferraro 2014-05-14 15:24:20 UTC
Added release note text.

Comment 10 Scott Mumford 2014-05-14 23:41:24 UTC
Refactored Doc Text into prose form.

Thanks Paul.


Note You need to log in before you can comment on or make changes to this bug.