Bug 1076258

Summary: [AS7] plugin's connection cleaner leaks classloader
Product: [Other] RHQ Project Reporter: Lukas Krejci <lkrejci>
Component: PluginsAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.11CC: hrupp
Target Milestone: GA   
Target Release: RHQ 4.11   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1077943 (view as bug list) Environment:
Last Closed: 2014-07-21 10:13:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1077943    

Description Lukas Krejci 2014-03-13 23:46:00 UTC 
Description of problem:

ASConnection has a static thread pool which runs connection "cleaners" to prevent connection leakage.

On every plugin container restart, the classes of the plugin are loaded anew, but the above thread pool is never shut down, leaving a thread running with a context classloader from the previous plugin container run.

This leads to an eventual OOME because of depleted perm gen memory needed to hold all the classes from all the plugin container runs in the agent's lifetime.

This problem has been made more apparent by our switch to HTTP client which is a much heavier library than the java's own Authenticator and HTTPUrlConnection we used before (but which had their own deal of problems).

Version-Release number of selected component (if applicable):
4.11-SNAPSHOT

How reproducible:
always

Steps to Reproduce:
1. inventory an AS7
2. periodically restart the plugin container without fully restarting the agent

Actual results:
out of memory error after a couple of repetitions (>10 usually, but depends on your perm gen set up)

Expected results:
no OOMEs

Additional info:
Comment 1 Lukas Krejci 2014-03-14 01:44:44 UTC 
commit 9d8b107414e926361ff2b7501e2094402de8e7f7
Author: Lukas Krejci <lkrejci>
Date:   Fri Mar 14 02:42:44 2014 +0100

    [BZ 1076258] Fix classloader leak in as7 plugin
    
    The plugin-wide thread pool for closing stale connections was leaving
    threads running and those threads held on to the "old" plugin classloaders
    from previous plugin container "runs".
    
    The AS7 plugin now has a plugin lifecycle listener that shuts down that
    thread pool preventing that leak from happening.
Comment 2 Heiko W. Rupp 2014-07-21 10:13:34 UTC 
Bulk closing of RHQ 4.11 issues, now that RHQ 4.12 is out.

If you find an issue with those, please open a new BZ, linking to the old one.