Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1031199 - EJB backing cache's can generate large retention from cancelled tasks in its scheduled executor's DelayedWorkQueue
EJB backing cache's can generate large retention from cancelled tasks in its ...
Status: CLOSED CURRENTRELEASE
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: EJB (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity unspecified
: DR0
: EAP 6.3.0
Assigned To: Paul Ferraro
Jan Martiska
Russell Dickenson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-15 16:19 EST by Aaron Ogburn
Modified: 2014-07-04 01:58 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The cache implementation for @Stateful EJBs in JBoss EAP 6 utilizes a scheduled executor for handling @StatefulTimeout logic. When a bean is accessed, it's previous timeout job is cancelled, and a new one is rescheduled when the invocation completes. By default, cancelling a task from an executor did not remove it from the queue. This caused a gradual memory leak, as cancelled tasks remain in the queue. In this release, the scheduled executor can be configured to remove the task from the queue upon cancellation. This avoids the memory leak.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-28 11:27:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker WFLY-2521 Major Resolved EJB backing cache's can generate large retention from cancelled tasks in its scheduled executor's DelayedWorkQueue 2016-11-30 00:24 EST

  None (edit)
Description Aaron Ogburn 2013-11-15 16:19:33 EST
Description of problem:

EJB backing cache's frequently cancel remove and passivation tasks with each access and replace them with new ones.  Per http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledThreadPoolExecutor.html cancelled tasks are not removed from queue until their scheduled delay passes.

So due to the java ScheduledThreadPoolExecutor's lazy cancelled task removal, this scheduled task cancellation and recreation model can potentially churn up quite a number of queued cancelled tasks sitting in the executor's DelayedWorkQueue.  With longer timeouts and frequent ejb access, this can generate substantial heap overhead.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. configure long ejb timeout with the Nonpassivating or PassivatingBackingCache
2. put load on the ejb

Actual results:

Cancelled tasks can pile in the executor's DelayedWorkQueue.


Expected results:

Cancelled tasks are flushed out of the executor's DelayedWorkQueue more eagerly.


Additional info:

It should be pretty easy to help limit any such build up by calling purge() [1] on the scheduled executor.  Likely don't want to purge with each cancel, so perhaps a purge() could be called on a configurable time delay or after a configurable amount of cancels?

[1] http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ThreadPoolExecutor.html#purge%28%29
Comment 1 JBoss JIRA Server 2013-11-18 17:42:54 EST
Paul Ferraro <paul.ferraro@redhat.com> updated the status of jira WFLY-2521 to Coding In Progress
Comment 2 JBoss JIRA Server 2013-11-18 17:53:34 EST
David Lloyd <david.lloyd@redhat.com> made a comment on jira WFLY-2521

Using STPE for this probably isn't a great fit as it just doesn't do well with lots of cancellations.  A simple sorted fast-access queue of some sort would probably work much better.
Comment 3 JBoss JIRA Server 2013-11-18 18:09:33 EST
Paul Ferraro <paul.ferraro@redhat.com> made a comment on jira WFLY-2521

[~dmlloyd] That's a good point.  It looks like queue removal is an O(N) operation - so even setRemoveOnCancelPolicy(true) is not ideal.  I'll open a separate jira to address this.
Comment 4 JBoss JIRA Server 2013-11-18 18:21:35 EST
Paul Ferraro <paul.ferraro@redhat.com> made a comment on jira WFLY-2521

See: https://issues.jboss.org/browse/WFLY-2534
Comment 5 Paul Ferraro 2013-11-19 12:18:53 EST
https://github.com/jbossas/jboss-eap/pull/706
Comment 6 Aaron Ogburn 2013-11-19 14:20:46 EST
Hey Paul,


Were you planning to add similar cancelled task removes in the NonPassivatingBackingCache's executor as well?


Thanks,

Aaron
Comment 7 Paul Ferraro 2013-11-20 10:07:07 EST
Good catch.  I've updated the PR.
Comment 8 Jan Martiska 2014-02-24 08:21:49 EST
Verified in 6.3.0.DR0.

Note You need to log in before you can comment on or make changes to this bug.