Created attachment 1367350 [details] foreman-proxy memory increase in graphana Description of problem: The capsule info show page (e.g. https://sat.example.com/smart_proxies/3-earqoasr03-example-com) triggers ajax call on load to fetch info about number of puppet modules and environments on the capsule - this involves iterating / scanning on every environment on the capsule and caching. Given the large number of environments, this present a significant load and is causing smart-proxy process to exhaust all system memory. Version-Release number of selected component (if applicable): Sat 6.2.11 How reproducible: 100% Reproducible with a capsule running 32GB RAM and 4000 puppet environments. Size of all puppet environments is around 50GB Steps to Reproduce: 1. Configure a capsule with a big number of puppet environments. In my case 4159 environments on a 32GB memory capsule 2. Set foreman-proxy log to DEBUG 3. Access to capsule info show page 4. Check that several GET requests are triggered on proxy.log GET /puppet/environments HTTP/1.1 GET /pulpnode/status/disk_usage GET /puppet/environments/production/classes 5. foreman-proxy start to scan all directories in alphabetical order: D, [2017-12-12T14:17:11.536456 #13791] DEBUG -- : Running scan_directory on _temp_10194_example_com: /etc/puppet/environments/_temp_10194_xxx_es/modules D, [2017-12-12T14:17:11.536913 #13791] DEBUG -- : Scanning concat classes in /etc/puppet/environments/_temp_10194_example_com/modules/concat/manifests/fragment.pp I, [2017-12-12T14:17:11.537262 #13791] INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf D, [2017-12-12T14:17:11.572148 #13791] DEBUG -- : Scanning concat classes in /etc/puppet/environments/_temp_10194_example_com/modules/concat/manifests/init.pp I, [2017-12-12T14:17:11.572546 #13791] INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf D, [2017-12-12T14:17:11.638181 #13791] DEBUG -- : Scanning concat classes in /etc/puppet/environments/_temp_10194_example_com/modules/concat/manifests/setup.pp I, [2017-12-12T14:17:11.638578 #13791] INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf D, [2017-12-12T14:17:11.664910 #13791] DEBUG -- : Scanning easy_type classes in /etc/puppet/environments/_temp_10194_example_com/modules/easy_type/manifests/init.pp I, [2017-12-12T14:17:11.665280 #13791] INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf D, [2017-12-12T14:17:11.783869 #13791] DEBUG -- : Scanning filemapper classes in /etc/puppet/environments/_temp_10194_example_com/modules/filemapper/manifests/init.pp I, [2017-12-12T14:17:11.784218 #13791] INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf Actual results: foreman-proxy starts to consume more and more memory until reaches swap. Probably it is caching modules and classes. See picture attached. Expected results: foreman-proxy do not consume all capsule memory available. Additional info: There is a workaround that prevents scanning / initializing all puppet classes and enviroments when accessing capsule info webpage. Basically consists on comment line 6 /usr/share/foreman/app/services/proxy_status/puppet.rb: https://github.com/theforeman/foreman/blob/develop/app/services/proxy_status/puppet.rb#L6 However, this workaround would be lost after an update
Please note that if Puppet 4.4 and higher is used, smart-proxy's puppet module will be using puppet's "environment-classes" api instead of built-in puppet class parser. It will, however, attempt to cache classes on startup, something that may need to be made configurable.
Created redmine issue http://projects.theforeman.org/issues/25984 from this bug
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.