Description of problem: Comparing 5.6.0.8 vs 5.5.4.2 idle memory usage 5.6.0.8 - 3501.72 5.5.4.2 - 3123.86 Version-Release number of selected component (if applicable): 5.6.0.8 How reproducible: Always Steps to Reproduce: 1. Turn on all roles except (git_owner and websocket) 2. 3. Actual results: Expected results: Additional info: Memory Usage Compared under workload to follow.
Heap info for retained objects relevant to message catalogs on 5.5.4.2 m: ruby's "guess" at memory size c: count of objects of this type at this location m: 961,119 | c: 9,507 | String ./ruby-2.2.5/lib/ruby/2.2.0/racc/parser.rb:258 m: 613,785 | c: 10,013 | String poparser.ry:139 m: 570,334 | c: 10,013 | String poparser.ry:219 m: 499,920 | c: 4 | FastGettext::GetText::MOFile ./ruby/2.2.5/gems/fast_gettext-1.0.0/lib/fast_gettext/po_file.rb:14 Heap info for retained objects relevant to message catalogs on 5.6.0.7 m: ruby's "guess" at memory size c: count of objects of this type at this location m: 1,123,649 | c: 10,749 | String ./ruby-2.2.5/lib/ruby/2.2.0/racc/parser.rb:258 m: 1,418,546 | c: 20,182 | String poparser.ry:139 m: 1,261,765 | c: 20,182 | String poparser.ry:219 m: 1,011,760 | c: 4 | FastGettext::GetText::MOFile ./ruby/2.2.5/gems/fast_gettext-1.0.0/lib/fast_gettext/po_file.rb:14 Note, in both cases, we have 4 FastGettext::GetText::MOFile objects, but that in 5.6.0.7, the size and count of the strings created are much larger. The 4 objects are because we're eager loading all of the locales we have po files for. Also, racc/parser is relevant here because fastgettext uses racc/parser to parse the po files. Note config/routes.rb (the rails app's routes) also are parser using racc/parser so take the changes to the allocations in racc/parser land with a grain of salt since our routes in 5.6 are also larger than 5.5. Related: git diff --shortstat 5.5.4.2 5.6.0.7 config/locales 7 files changed, 138706 insertions(+), 27305 deletions(-) We have added a net of 111,000+ lines to all of the po files from 5.5.4.2 to 5.6.0.7. We can test the effect of lazy loading the 4 message catalogs (loading only the english one at boot most of the time) by removing the non-english locales. Additionally, upstream fastgettext now support lazy loading po/mo files so I put this PR together: https://github.com/ManageIQ/manageiq/pull/8525
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/b55c210b50d2ac5d672753cef6c0d7f1462dd086 commit b55c210b50d2ac5d672753cef6c0d7f1462dd086 Author: Milan Zazrivec <mzazrivec> AuthorDate: Thu Jun 2 16:04:13 2016 +0200 Commit: Joe Rafaniello <jrafanie> CommitDate: Thu Jun 2 11:04:12 2016 -0400 Create list of human locale names from a pre-prepared yaml file Changes: 1. A new rake task: gettext:extract_locale_names to create list of human locale names, as they are presented in existing language catalogs. The list is saved as config/human_locale_names.yaml 2. During application start, the list of human locale names is taken from the config/human_locale_names.yaml file. This helps to speed things up during start, as we don't have to (lazy) load every gettext catalog during when all we need is one string. https://bugzilla.redhat.com/show_bug.cgi?id=1341321 config/human_locale_names.yaml | 6 ++++++ config/initializers/fast_gettext.rb | 1 + lib/tasks/gettext.rake | 22 ++++++++++++++++++++++ lib/vmdb/fast_gettext_helper.rb | 21 +++------------------ 4 files changed, 32 insertions(+), 18 deletions(-) create mode 100644 config/human_locale_names.yaml
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/e023d550e58949d6acdcfb0a65797af09ee13d61 commit e023d550e58949d6acdcfb0a65797af09ee13d61 Author: Joe Rafaniello <jrafanie> AuthorDate: Fri May 6 18:07:08 2016 -0400 Commit: Joe Rafaniello <jrafanie> CommitDate: Thu Jun 2 11:04:06 2016 -0400 Lazy load message catalogs (po) on first use Use fast_gettext 1.1.0 containing: https://github.com/grosser/fast_gettext/pull/79 https://bugzilla.redhat.com/show_bug.cgi?id=1341321 Gemfile | 1 + lib/vmdb/fast_gettext_helper.rb | 5 +++++ 2 files changed, 6 insertions(+)
Lazy loading the message catalogs in fastgettext [1] should get memory usage much lower, preliminary estimates are ~15 MB per rails process on appliances. Our changes to make sure we don't load all 4 catalogs in [2] make it possible to only load one of the catalogs. [1] https://github.com/grosser/fast_gettext/pull/79 [2] https://github.com/ManageIQ/manageiq/pull/8525
There are more improvements on the horizon but this should get us much closer to where we want.
https://github.com/ManageIQ/manageiq/pull/9133
https://github.com/ManageIQ/manageiq/pull/7620
previous: 5.6.0.8 - 3501.72 5.5.4.2 - 3123.86 now: 5.7.0.0 - 2830.29 much lower, it seems.
Hi Archit, Is this comment 11 with the same number and types of workers running? thanks
I saw this come into my inbox so I pulled up my old data from this and 5.7 eliminated the replication worker which was turned on in previous versions of this test which makes the above comparison effectively null. Archit could you re-run this and compare with the old version with the same roles and workers. Additionally, using the idle workload in cfme-performance will give a repeatable result/measurement that will be comparable.
Hi Joe, Thanks for pointing it out. It seems I had made a minor error. I've run this again and got this on 5700 (without any database loaded): 2396.43 The worker count is as per default settings. Appliance Roles: (all except git_owner and websocket): - automate, database_operations, database_synchronization, ems_inventory, ems_metrics_collector, ems_metrics_coordinator, ems_metrics_processor, ems_operations, event, notifier, reporting, rhn_mirror, scheduler, smartproxy, smartstate, user_interface, web_services