Bug 1077943 - [AS7] plugin's connection cleaner leaks classloader
Summary: [AS7] plugin's connection cleaner leaks classloader
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Plugin -- JBoss EAP 6
Version: JON 3.1.2,JON 3.2
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ER03
: JON 3.3.0
Assignee: Lukas Krejci
QA Contact: Filip Brychta
URL:
Whiteboard:
Keywords: Regression, TestCaseNeeded
: 1119516 (view as bug list)
Depends On: 1076258 1119516
Blocks: 1079070 1145561
TreeView+ depends on / blocked
 
Reported: 2014-03-18 23:19 UTC by Larry O'Leary
Modified: 2018-12-05 17:45 UTC (History)
5 users (show)

(edit)
The ASConnection of the EAP plug-in has a static thread pool which runs connection "cleaners" to prevent connection leakage.

When the plug-in container was restarted, the classes of the plug-in were loaded again. However the static thread pool was never shut down which left a thread running with a context classloader from the previous plug-in container.

The stale thread lead to an eventual out-of-memory exception because of depleted permgen memory needed to hold all the classes from all the plug-in container runs in the agent's lifetime.

The AS7 plug-in now has a plug-in lifecycle listener that shuts down the aforementioned thread pool preventing the leak from happening.
Clone Of: 1076258
: 1079070 1119516 (view as bug list)
(edit)
Last Closed: 2014-12-11 13:59:15 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 615377 None None None Never
Red Hat Knowledge Base (Solution) 761123 None None None Never

Internal Trackers: 615377

Description Larry O'Leary 2014-03-18 23:19:00 UTC
+++ This bug was initially created as a clone of Bug #1076258 +++

Description of problem:

ASConnection has a static thread pool which runs connection "cleaners" to prevent connection leakage.

On every plugin container restart, the classes of the plugin are loaded anew, but the above thread pool is never shut down, leaving a thread running with a context classloader from the previous plugin container run.

This leads to an eventual OOME because of depleted perm gen memory needed to hold all the classes from all the plugin container runs in the agent's lifetime.

This problem has been made more apparent by our switch to HTTP client which is a much heavier library than the java's own Authenticator and HTTPUrlConnection we used before (but which had their own deal of problems).

Version-Release number of selected component (if applicable):
4.11-SNAPSHOT

How reproducible:
always

Steps to Reproduce:
1. inventory an AS7
2. periodically restart the plugin container without fully restarting the agent

Actual results:
out of memory error after a couple of repetitions (>10 usually, but depends on your perm gen set up)

Expected results:
no OOMEs

Additional info:

--- Additional comment from Lukas Krejci on 2014-03-13 21:44:44 EDT ---

commit 9d8b107414e926361ff2b7501e2094402de8e7f7
Author: Lukas Krejci <lkrejci@redhat.com>
Date:   Fri Mar 14 02:42:44 2014 +0100

    [BZ 1076258] Fix classloader leak in as7 plugin
    
    The plugin-wide thread pool for closing stale connections was leaving
    threads running and those threads held on to the "old" plugin classloaders
    from previous plugin container "runs".
    
    The AS7 plugin now has a plugin lifecycle listener that shuts down that
    thread pool preventing that leak from happening.

Comment 6 Simeon Pinder 2014-07-31 15:49:44 UTC
Moving to ON_QA as available to test with brew build of DR01: https://brewweb.devel.redhat.com//buildinfo?buildID=373993

Comment 8 Lukas Krejci 2014-09-10 12:43:51 UTC
BZ 1119516 is ON_QA, so this can, too, be retested. Moving to ON_QA again.

Comment 9 Jan Bednarik 2014-09-17 15:37:38 UTC
After repeatedly restarting the plugin container the PermGen used memory keeps growing, thus the class loader leak is still visible. After consulting this issue with devs it seems that the problem might not be specifically in EAP's classloader but rather in other component. The issue is covered by BZ 1134902, thus adding 'depends on' flag.

Comment 10 Filip Brychta 2014-09-22 15:31:48 UTC
Verified on
Version :	
3.3.0.ER03
Build Number :	
4aefe39:44e33a4

Scenario:
1- AS7 is in inventory
2- schedule agent operation "Restart Plugin Container" each 2 minutes

Result:
After 6 hours which corresponds to ~ 180 Plugin container restarts is the perm gen still stable


This verifies leaks in AS7 plugin. There are other leaks which are covered by bz 1134902

Comment 11 Filip Brychta 2015-06-04 14:34:42 UTC
*** Bug 1119516 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.