Augeas libraries available in Agent installation
+++ This bug was initially created as a clone of Bug #721151 +++
+++ This bug was initially created as a clone of Bug #721150 +++
Agent JVM is using large amounts of system memory even when Java heap is relatively small. This results in slow system performance and in some instances, the agent's JVM being terminated by the Linux kernel.
This memory usage is outside of the JVM itself and appears to occur in a native library used by the agent's JVM. The library in use in all reported instances of the issue is libaugeas.so.
The high memory usage is due to a plug-ins construction of net.augeas.Augeas without a corresponding Augeas.close() call when the object is no longer needed. This results in the underlying native library hanging onto resources that have been initialized/requested with the construction of the Augeas object.
Memory usage varies depending on the number of Apache instances that are on the agent host machine.
This issue occurs even if there are no Apache instances in inventory. This is because the Apache plug-in utilizes the Augeas libraries, if available, when attempting discovery, which by default is executed every 15 minutes.
Furthermore, if there are Apache instances in inventory, the leak occurs at a faster rate because instances of Augeas are created during service scans and configuration syncs.
This issue affects the Augeas and Apache plug-ins and potentially others. Essentially, anywhere that we are creating a new instance of net.augeas.Augeas without explicitly calling Augeas.close() after we are done with the Augeas libraries.
As suggested by Mike, we should probably be using some type of design pattern to ensure that the Augeas bindings can not be used directly but instead everything go through AugeasProxy or some other delegate that could help reduce this potential.
On that note, any object that requires the ability to use Augeas as a data field should also override the finalize() method to explicitly close the Augeas instance during garbage collection.
This issue may also have been originally identified in https://bugzilla.redhat.com/show_bug.cgi?id=589674 however it does not appear that it was directly linked to Augeas or the use of Apache however, the description and the comments in the bug are consistent to this issue. Looking at the Apache plug-in in the 3.0.0 release, it required the native library support to be enabled to start the Apache resources and perform server discovery. So, by disabling native support, you are essentially also disabling the use of the Augeas libs which would result in the issue going away.
Although we really need to properly address the use of the Augeas libraries, the Augeas Java Binding should at minimum clean up these resource leaks by using a finalizer which performs the cleanup in the event that the caller fails to do it. I logged the following two Augeas bugs to address this in the Augeas Java Binding:
With Filip to fix in master
Author: Lukas Krejci <firstname.lastname@example.org>
Date: Mon Aug 1 14:11:28 2011 +0200
BZ 721152 - fixing the augeas memory leak - cherrypicked over from release-3.0.0
while verifying this, encountered https://bugzilla.redhat.com/show_bug.cgi?id=728292
Lukas could you please help me by providing steps to verify this BZ
This is a hard one to test manually.
The steps would be:
1) Configure the agent to run configuration checks frequently (say every 5
1) Inventorize an apache instance
2) Enable augeas in the connection settings
3) let it run overnight
4) no drastic memory increase of the agent process should be visible
Verified on build# 485 (Version: 4.1.0-SNAPSHOT Build Number: e07065d)
Configured the agent to run config checks every 5 minutes. Inventoried apache instance and enabled augeas in connection settings.
After the overnight run observed that the agent process in top command does not show any drastic memory increase. (Earlier it was around 25 to 26% and now it is around 26 to 27%)
changing status of VERIFIED BZs for JON 2.4.2 and JON 3.0 to CLOSED/CURRENTRELEASE