Bug 1525514

Summary: foreman-proxy high memory usage when scanning a big number of puppet environments directories
Product: Red Hat Satellite Reporter: Alberto Losada <alosadag>
Component: Foreman ProxyAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.2.11CC: alosadag, apatel, ddolguik, dmitri, inecas, kgaikwad, oprazak
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-01 18:35:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
foreman-proxy memory increase in graphana none

Description Alberto Losada 2017-12-13 13:15:15 UTC 
Created attachment 1367350 [details]
foreman-proxy memory increase in graphana

Description of problem:

The capsule info show page (e.g. https://sat.example.com/smart_proxies/3-earqoasr03-example-com) triggers ajax call on load to fetch info about number of puppet modules and environments on the capsule - this involves iterating / scanning on every environment on the capsule and caching. Given the large number of environments, this present a significant load and is causing smart-proxy process to exhaust all system memory. 

Version-Release number of selected component (if applicable):

Sat 6.2.11

How reproducible:

100% Reproducible with a capsule running 32GB RAM and 4000 puppet environments.
Size of all puppet environments is around 50GB

Steps to Reproduce:

1. Configure a capsule with a big number of puppet environments. In my case 4159 environments on a 32GB memory capsule
2. Set foreman-proxy log to DEBUG
3. Access to capsule info show page
4. Check that several GET requests are triggered on proxy.log
     GET /puppet/environments HTTP/1.1
     GET /pulpnode/status/disk_usage
     GET /puppet/environments/production/classes
5. foreman-proxy start to scan all directories in alphabetical order:

D, [2017-12-12T14:17:11.536456 #13791] DEBUG -- : Running scan_directory on _temp_10194_example_com: /etc/puppet/environments/_temp_10194_xxx_es/modules
D, [2017-12-12T14:17:11.536913 #13791] DEBUG -- : Scanning concat classes in /etc/puppet/environments/_temp_10194_example_com/modules/concat/manifests/fragment.pp
I, [2017-12-12T14:17:11.537262 #13791]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
D, [2017-12-12T14:17:11.572148 #13791] DEBUG -- : Scanning concat classes in /etc/puppet/environments/_temp_10194_example_com/modules/concat/manifests/init.pp
I, [2017-12-12T14:17:11.572546 #13791]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
D, [2017-12-12T14:17:11.638181 #13791] DEBUG -- : Scanning concat classes in /etc/puppet/environments/_temp_10194_example_com/modules/concat/manifests/setup.pp
I, [2017-12-12T14:17:11.638578 #13791]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
D, [2017-12-12T14:17:11.664910 #13791] DEBUG -- : Scanning easy_type classes in /etc/puppet/environments/_temp_10194_example_com/modules/easy_type/manifests/init.pp
I, [2017-12-12T14:17:11.665280 #13791]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
D, [2017-12-12T14:17:11.783869 #13791] DEBUG -- : Scanning filemapper classes in /etc/puppet/environments/_temp_10194_example_com/modules/filemapper/manifests/init.pp
I, [2017-12-12T14:17:11.784218 #13791]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf


Actual results:

foreman-proxy starts to consume more and more memory until reaches swap. Probably it is caching modules and classes. See picture attached. 

Expected results:

foreman-proxy do not consume all capsule memory available.

Additional info:

There is a workaround that prevents scanning / initializing all puppet classes and enviroments when accessing capsule info webpage. Basically consists on comment line 6 /usr/share/foreman/app/services/proxy_status/puppet.rb: https://github.com/theforeman/foreman/blob/develop/app/services/proxy_status/puppet.rb#L6

However, this workaround would be lost after an update
Comment 4 Dmitri Dolguikh 2018-02-28 21:39:39 UTC 
Please note that if Puppet 4.4 and higher is used, smart-proxy's puppet module will be using puppet's "environment-classes" api instead of built-in puppet class parser. 

It will, however, attempt to cache classes on startup, something that may need to be made configurable.
Comment 5 Ondřej Pražák 2019-02-05 09:49:49 UTC 
Created redmine issue http://projects.theforeman.org/issues/25984 from this bug
Comment 6 Bryan Kearney 2019-07-02 18:02:04 UTC 
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.
Comment 7 Bryan Kearney 2019-08-01 18:35:29 UTC 
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.