Bug 1123970 - Use of augeas results in memory leak that eventually brings down the agent and/or its platform
Summary: Use of augeas results in memory leak that eventually brings down the agent an...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: JBoss Enterprise Web Server 2
Classification: JBoss
Component: JON Plugin
Version: 2.0.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Jean-frederic Clere
QA Contact: Michal Karm Babacek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-28 18:33 UTC by Larry O'Leary
Modified: 2018-12-06 17:30 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
This release of JBoss EWS carries a known issue that causes a memory leak. The leak presents if an Apache Web Server resource has its <parameter>Enable augeas support</parameter> connection setting set to <literal>yes</literal>, the agent's memory footprint continues to grow until eventually the process is termninated by the kernel. This issue is expected to be resolved in future release of the product, but a workaround is to disable the use of Augeas for the httpd resource.
Clone Of:
Environment:
Last Closed: 2017-10-10 15:54:31 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 721152 0 urgent CLOSED Plug-ins initializing Augeas without explicitly de-initializing is resulting in large memory and resource leak 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 1133373 0 None None None Never

Internal Links: 721152

Description Larry O'Leary 2014-07-28 18:33:23 UTC
Description of problem:
If an Apache Web Server resource has its _Enable augeas support_ connection setting set to _yes_, the agent's memory footprint continues to grow:

    Fri Jul 25 19:57:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  4.4  5.8 1622516 228252 pts/0  Sl   19:32   1:04 /etc/alternative
    Fri Jul 25 19:58:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  4.9  5.9 1622516 234088 pts/0  Sl   19:32   1:14 /etc/alternative
    Fri Jul 25 19:59:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  5.3  6.2 1622516 243852 pts/0  Sl   19:32   1:23 /etc/alternative
    Fri Jul 25 20:00:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  5.7  6.5 1622516 255784 pts/0  Sl   19:32   1:32 /etc/alternative
    Fri Jul 25 20:01:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  6.0  6.8 1622516 268380 pts/0  Sl   19:32   1:42 /etc/alternative
    Fri Jul 25 20:02:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  6.4  7.2 1688052 285176 pts/0  Sl   19:32   1:52 /etc/alternative
    Fri Jul 25 20:03:01 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615  6.8  7.4 1688052 292884 pts/0  Sl   19:32   2:03 /etc/alternative
    ...
    Fri Jul 25 23:35:20 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615 15.8 61.1 4113412 2400304 pts/0 Sl   19:32  38:24 /etc/alternative
    Fri Jul 25 23:36:21 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615 15.9 61.7 4113412 2421996 pts/0 Sl   19:32  38:44 /etc/alternative
    Fri Jul 25 23:37:21 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615 15.9 62.0 4244484 2435712 pts/0 Sl   19:32  38:56 /etc/alternative
    Fri Jul 25 23:38:22 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    jonadmin  2615 15.9 62.3 4244484 2447460 pts/0 Sl   19:32  39:07 /etc/alternative
    Fri Jul 25 23:39:22 EDT 2014
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

Eventually, the process is terminated by the kernel due to its excessive use of memory.

Version-Release number of selected component (if applicable):
EWS 2.0.1 + EWS Plug-in Pack for JBoss ON 3.2

How reproducible:
Always

Steps to Reproduce:
1.  Install and start JBoss EWS 2.0.1 HTTPD for RHEL 6 x86_64.
2.  Install and start JBoss ON 3.2 system.
3.  Import _Apache HTTP Server_ into inventory.
4.  From the imported _Apache HTTP Server's_ connection settings page, _Enable augeas support_ to _Yes_.
5.  From platform resource, invoke the _manual discovery_ operation to repeat every 30 seconds for indefinitely.
6.  Monitor the VSZ and RSS memory usage of the JBoss ON agent JVM over time:

        while true; do date; ps u -p $(ps -ef | grep AgentMain | grep java | awk '{print $2}'); sleep 60s; done

Actual results:
Every minute, RSS and VSZ memory increase. Depending on total available memory on the machine, the process may eventually be killed and the following will be reported in /var/log/messages:

    Jul 25 23:38:48 vm204 kernel: Out of memory: Kill process 2615 (java) score 614 
or sacrifice child
    Jul 25 23:38:48 vm204 kernel: Killed process 2615, UID 502, (java) total-vm:4244
484kB, anon-rss:2458372kB, file-rss:2588kB

Expected results:
No memory growth and process doesn't get terminated.

Some memory growth is expected on first couple of executions of service discovery but it should be stable and VSZ should not grow.

Additional info:
This issue is related to service discovery on the Apache Web Server resource. If you disable the use of Augeas for the said httpd resource, memory is stable. Additionally, if you enable it and disable service discovery, memory is stable. Server discovery DOES NOT cause the memory growth. 

In my test instance, I am using the default configuration from EWS HTTPD (httpd.conf).

It isn't clear if it is the Directory resource type or the IfModule resource type or both of them that is causing this issue.

Comment 1 Larry O'Leary 2014-07-28 20:55:54 UTC
This seems similar to what was fixed in BZ-721152. A regression perhaps? Or just a missed test-case?

Comment 2 Jean-frederic Clere 2014-07-29 07:10:56 UTC
Is the same problem existing with RHQ master too?

Comment 3 Larry O'Leary 2014-07-29 14:05:25 UTC
Actually, I am not certain. The plug-in was from JBoss ON 3.2 GA release from back in December 2013. Do you think there has been changes here? If so, I can retest using an updated build.

Comment 4 Jean-frederic Clere 2014-07-29 16:54:29 UTC
No need to retest...
I have checked no changes (except on windows) since December 2013.


Note You need to log in before you can comment on or make changes to this bug.