Description of problem:
Memory Leak in the main "MIQ Server" process over a extended period of time.
Version-Release number of selected component (if applicable): 5.9.0
How reproducible:
Run any server over a large stretch of time, and the memory will slowly leak. More workers on the same box will increase the rate at witch it leaks.
Steps to Reproduce:
1. Boot an appliance
2. Watch it leak (takes a few days)
If you wish to speed up this process, you can run the following from the /var/www/miq/vmdb directory:
$ bin/rails runner "cfg = MiqServer.my_server.get_config; cfg.config[:server][:monitor_poll] = '1.second'; cfg.config[:server][:worker_monitor_frequency] = '1.second'; MiqServer.my_server.set_config cfg"
Actual results:
Server leaks around 10MB every 10 minutes or so, depending on the number of workers configured for that appliance.
Expected results:
No Leak.
We have a set of fixes already in place for this issue:
https://github.com/ManageIQ/manageiq/pull/16837https://github.com/ManageIQ/manageiq-api/pull/288https://github.com/ManageIQ/manageiq-automation_engine/pull/146https://github.com/ManageIQ/manageiq-ui-classic/pull/3266
Turned out to be a small, unknown leak in ruby that was prevalent in our application since we defer loading a lot of libraries until they are needed using `require` inside of a method call, and that we were using `Pathnames` in our Rails `autoload_path` that was eventually getting added to the `$LOAD_PATH` in ruby.
This would make it so that every `require` call would leak a small amount of memory, and would never be released. This only happens when Pathnames are used instead of raw strings in the `$LOAD_PATH`, so since both are valid, switching to using raw strings is the workaround we are using to prevent the leak.
Description of problem: Memory Leak in the main "MIQ Server" process over a extended period of time. Version-Release number of selected component (if applicable): 5.9.0 How reproducible: Run any server over a large stretch of time, and the memory will slowly leak. More workers on the same box will increase the rate at witch it leaks. Steps to Reproduce: 1. Boot an appliance 2. Watch it leak (takes a few days) If you wish to speed up this process, you can run the following from the /var/www/miq/vmdb directory: $ bin/rails runner "cfg = MiqServer.my_server.get_config; cfg.config[:server][:monitor_poll] = '1.second'; cfg.config[:server][:worker_monitor_frequency] = '1.second'; MiqServer.my_server.set_config cfg" Actual results: Server leaks around 10MB every 10 minutes or so, depending on the number of workers configured for that appliance. Expected results: No Leak.