Bug 1535720 - Memory Leak in MiqServer process
Summary: Memory Leak in MiqServer process
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.9.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: GA
: 5.10.0
Assignee: Nick LaMuro
QA Contact: Dave Johnson
URL:
Whiteboard:
: 1511136 1511859 1511890 (view as bug list)
Depends On:
Blocks: 1536658 1536672 1536692
TreeView+ depends on / blocked
 
Reported: 2018-01-17 23:59 UTC by Nick LaMuro
Modified: 2018-10-16 16:12 UTC (History)
8 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1536658 1536672 1536692 (view as bug list)
Environment:
Last Closed: 2018-06-21 20:28:59 UTC
Category: ---
Cloudforms Team: CFME Core


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3618221 None None None 2018-10-16 16:12:22 UTC

Description Nick LaMuro 2018-01-17 23:59:08 UTC
Description of problem:

Memory Leak in the main "MIQ Server" process over a extended period of time.


Version-Release number of selected component (if applicable): 5.9.0


How reproducible:

Run any server over a large stretch of time, and the memory will slowly leak.  More workers on the same box will increase the rate at witch it leaks.


Steps to Reproduce:
1. Boot an appliance
2. Watch it leak (takes a few days)

If you wish to speed up this process, you can run the following from the /var/www/miq/vmdb directory:

    $ bin/rails runner "cfg = MiqServer.my_server.get_config; cfg.config[:server][:monitor_poll] = '1.second'; cfg.config[:server][:worker_monitor_frequency] = '1.second'; MiqServer.my_server.set_config cfg"



Actual results:

Server leaks around 10MB every 10 minutes or so, depending on the number of workers configured for that appliance.


Expected results:

No Leak.

Comment 2 Nick LaMuro 2018-01-18 00:04:43 UTC
We have a set of fixes already in place for this issue:

https://github.com/ManageIQ/manageiq/pull/16837
https://github.com/ManageIQ/manageiq-api/pull/288
https://github.com/ManageIQ/manageiq-automation_engine/pull/146
https://github.com/ManageIQ/manageiq-ui-classic/pull/3266


Turned out to be a small, unknown leak in ruby that was prevalent in our application since we defer loading a lot of libraries until they are needed using `require` inside of a method call, and that we were using `Pathnames` in our Rails `autoload_path` that was eventually getting added to the `$LOAD_PATH` in ruby.

This would make it so that every `require` call would leak a small amount of memory, and would never be released.  This only happens when Pathnames are used instead of raw strings in the `$LOAD_PATH`, so since both are valid, switching to using raw strings is the workaround we are using to prevent the leak.

Comment 6 dmetzger 2018-02-23 13:42:19 UTC
*** Bug 1511859 has been marked as a duplicate of this bug. ***

Comment 7 dmetzger 2018-02-23 13:44:28 UTC
*** Bug 1511890 has been marked as a duplicate of this bug. ***

Comment 8 Joe Rafaniello 2018-02-28 18:25:57 UTC
*** Bug 1511136 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.