This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 721152 - Plug-ins initializing Augeas without explicitly de-initializing is resulting in large memory and resource leak
Plug-ins initializing Augeas without explicitly de-initializing is resulting ...
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Plugins (Show other bugs)
4.2
All Linux
urgent Severity urgent (vote)
: ---
: ---
Assigned To: Lukas Krejci
Mike Foley
:
Depends On: 721150 721151
Blocks: jon3 rhq41beta
  Show dependency treegraph
 
Reported: 2011-07-13 17:09 EDT by Larry O'Leary
Modified: 2014-07-28 16:55 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 721151
Environment:
Apache httpd Apache Plug-in Augeas libraries available in Agent installation Linux platform
Last Closed: 2012-02-07 14:28:25 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Larry O'Leary 2011-07-13 17:09:39 EDT
+++ This bug was initially created as a clone of Bug #721151 +++

+++ This bug was initially created as a clone of Bug #721150 +++

Agent JVM is using large amounts of system memory even when Java heap is relatively small. This results in slow system performance and in some instances, the agent's JVM being terminated by the Linux kernel.

This memory usage is outside of the JVM itself and appears to occur in a native library used by the agent's JVM. The library in use in all reported instances of the issue is libaugeas.so. 

The high memory usage is due to a plug-ins construction of net.augeas.Augeas without a corresponding Augeas.close() call when the object is no longer needed. This results in the underlying native library hanging onto resources that have been initialized/requested with the construction of the Augeas object.

Memory usage varies depending on the number of Apache instances that are on the agent host machine.

This issue occurs even if there are no Apache instances in inventory. This is because the Apache plug-in utilizes the Augeas libraries, if available, when attempting discovery, which by default is executed every 15 minutes.

Furthermore, if there are Apache instances in inventory, the leak occurs at a faster rate because instances of Augeas are created during service scans and configuration syncs.
Comment 1 Larry O'Leary 2011-07-13 17:15:01 EDT
This issue affects the Augeas and Apache plug-ins and potentially others. Essentially, anywhere that we are creating a new instance of net.augeas.Augeas without explicitly calling Augeas.close() after we are done with the Augeas libraries.

As suggested by Mike, we should probably be using some type of design pattern to ensure that the Augeas bindings can not be used directly but instead everything go through AugeasProxy or some other delegate that could help reduce this potential.

On that note, any object that requires the ability to use Augeas as a data field should also override the finalize() method to explicitly close the Augeas instance during garbage collection.
Comment 2 Larry O'Leary 2011-07-13 17:20:25 EDT
This issue may also have been originally identified in https://bugzilla.redhat.com/show_bug.cgi?id=589674 however it does not appear that it was directly linked to Augeas or the use of Apache however, the description and the comments in the bug are consistent to this issue. Looking at the Apache plug-in in the 3.0.0 release, it required the native library support to be enabled to start the Apache resources and perform server discovery. So, by disabling native support, you are essentially also disabling the use of the Augeas libs which would result in the issue going away.
Comment 3 Larry O'Leary 2011-07-19 19:43:25 EDT
Although we really need to properly address the use of the Augeas libraries, the Augeas Java Binding should at minimum clean up these resource leaks by using a finalizer which performs the cleanup in the event that the caller fails to do it. I logged the following two Augeas bugs to address this in the Augeas Java Binding:

https://fedorahosted.org/augeas/ticket/212
https://fedorahosted.org/augeas/ticket/213
Comment 4 Charles Crouch 2011-07-25 12:35:23 EDT
With Filip to fix in master
Comment 5 Lukas Krejci 2011-08-04 08:50:48 EDT
commit ba3fcbe5cdea5c384743c0a90e71e73a48707b9f
Author: Lukas Krejci <lkrejci@redhat.com>
Date:   Mon Aug 1 14:11:28 2011 +0200

    BZ 721152 - fixing the augeas memory leak - cherrypicked over from release-3.0.0
Comment 6 Mike Foley 2011-08-04 11:37:27 EDT
while verifying this, encountered https://bugzilla.redhat.com/show_bug.cgi?id=728292
Comment 7 Venkat 2011-09-26 08:34:48 EDT
Lukas could  you please help me by providing steps to verify this BZ
Comment 8 Lukas Krejci 2011-09-29 11:38:45 EDT
This is a hard one to test manually.

The steps would be:

1) Configure the agent to run configuration checks frequently (say every 5
minutes)
1) Inventorize an apache instance
2) Enable augeas in the connection settings
3) let it run overnight
4) no drastic memory increase of the agent process should be visible
Comment 9 Sunil Kondkar 2011-10-10 06:55:49 EDT
Verified on build# 485 (Version: 4.1.0-SNAPSHOT Build Number: e07065d)

Configured  the agent to run config checks every 5 minutes. Inventoried apache instance and enabled augeas in connection settings. 

After the overnight run observed that the agent process in top command does not show any drastic memory increase. (Earlier it was around 25 to 26% and now it is around 26 to 27%)
Comment 10 Mike Foley 2012-02-07 14:28:25 EST
changing status of VERIFIED BZs for JON 2.4.2 and JON 3.0 to CLOSED/CURRENTRELEASE

Note You need to log in before you can comment on or make changes to this bug.