Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
User story:
publishing or promoting a CV with puppet modules does consume up to 30GB of memory in sidekiq process.
The root cause is in this call flow:
- there is Actions::Katello::Foreman::ContentUpdate dynflow step in Finalize phase
- it calls ::Katello::Foreman.update_puppet_environment(content_view, environment)
- it calls PuppetClassImporter.new(..).update_environment
- it executes "changed = self.changes" from foreman
- and directly the method https://github.com/theforeman/foreman/blob/2.5-stable/app/services/puppet_class_importer.rb#L19 does consume so much memory
(the above statement "method 'changes' itself causes the high memory usage" was confirmed by running a rake script at a customer where a PuppetClassImporter(..).changes was the method that bumped memory usage by >4GB, _alone_ )
Further internal testing revealed that the scaling factor affecting the high memory usage is EnvironmentClass.count . For 120k EnvironmentClass items, rake script executing the "changes" method consumed 900MB memory (400MB rake itself, 500MB the method). For 260k EnvironmentClass items, I got 1552MB usage (400MB rake, 1152MB the method). Lineraly prolonging this to 1M EnvironmentClass items (that the customer has), we get 4GB memory per *one* execution of the method.
Due to the way how ruby does (not) return freed memory, repeated calling of the method by different PuppetClassImporter instances does consume further GBs of memory, every time. Until the usage is (probably) stabilised at 30GB per *one* sidekiq process.
Please optimise the foreman method (being used in 6.10 as well, hence we do support it there also) to be more memory efficient for scaled EnvironmentClass instances.
Version-Release number of selected component (if applicable):
Sat6.8 (same affects 6.9 and will 6.10)
How reproducible:
100%
Steps to Reproduce:
1. Import many puppet modules (I will provide a script for it)
2. Have any repo enabled
3. Create many CVs with many CV versions having many puppet modules associated each.
4. Monitor memory usage of sidekiq processes during the CV publish/promote
5. Monitor # of puppet env. classes:
su - postgres -c "psql foreman -c \"select count(*) from environment_classes;\""
Actual results:
memory usage grows linearly with the EnvironmentClass.count / count(*) from environment_classes. For 1M classes, memory bumps by 4GB in a CV publish/promote.
Expected results:
something more sustainable :)
Additional info:
Some stats of the impact of # of env.classes to RSS usage of the 'changes' method (RSS usage of rake running the method):
RSS EnvClass.count
878920 119903
1059592 129122
1034964 138341
1029156 147560
897756 156779
1010816 165998
1233568 175217
1187076 184436
1244400 193655
1244380 202874
1300772 212093
1269580 221312
1286400 230531
1316340 239750
1472676 248969
1551976 258188
1569808 267407
1578724 276626
.. (test continues) ..
(In reply to Melanie Falz from comment #3)
> The same customer hit the issue again and asked for a fix, is it possible to
> give an outlook on this?
Needinfo on satellite6-bugs is not maintained by anybody and it rather causes more harm than gain. But I am very uncertain who in particular to ask for - raising needinfo to a few people, please cancel them once either of them provides an answer.
Hello, sorry for the inconvenience.
This behavior seems to be the same since Satellite 6.0, so it's not a regression.
Adding it to our backlog and when we will have more capacity we will look into it.
This bugzilla only applies to Satellite 6.9 and prior, as the Puppet content type was not introduced with Pulp 3. Based upon this, I am going to proceed with closing this bugzilla. If there are any concerns with this action, please reach out to me. Thanks!