Bug 1448963

Summary: MiqVimBrokerWorker exceeding memory after upgrading from 5.6 -> 5.7
Product: Red Hat CloudForms Management Engine Reporter: Jared Deubel <jdeubel>
Component: ProvidersAssignee: Adam Grare <agrare>
Status: CLOSED CURRENTRELEASE QA Contact: luke couzens <lcouzens>
Severity: high Docs Contact:
Priority: high    
Version: 5.7.0CC: cpelland, jdeubel, jfrey, jhardy, jocarter, lcouzens, obarenbo, simaishi
Target Milestone: GAKeywords: TestOnly
Target Release: 5.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: upgrade:black
Fixed In Version: 5.9.0.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1449412 1450526 (view as bug list) Environment:
Last Closed: 2018-03-06 15:36:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1449412, 1450526    
Attachments:
Description Flags
5.7.0.17 revert of https://github.com/ManageIQ/manageiq/pull/11629 none

Description Jared Deubel 2017-05-08 21:27:48 UTC 
Description of problem:
MiqVimBrokerWorker exceeding memory after upgrading from 5.6 -> 5.7. To reduce memory usage retrieve host storage devices host by host instead of in one call.

This was originally done to increase performance but the vast majority of the overhead was due to running garbage collection on every API call.

Now that we don't do this the performance increase of retrieving all hosts in one call is greatly reduced. Want to revert these changes back so we don't consume as much memory.


Version-Release number of selected component (if applicable):
5.7.0.1717
Comment 3 Adam Grare 2017-05-09 01:33:54 UTC 
Created attachment 1277196 [details]
5.7.0.17 revert of https://github.com/ManageIQ/manageiq/pull/11629
Comment 4 CFME Bot 2017-05-09 02:13:32 UTC 
New commit detected on ManageIQ/manageiq-providers-vmware/master:
https://github.com/ManageIQ/manageiq-providers-vmware/commit/fa2049fb136b91b706fb3f887434003c1642d41d

commit fa2049fb136b91b706fb3f887434003c1642d41d
Author:     Adam Grare <agrare>
AuthorDate: Tue Mar 14 20:00:13 2017 -0400
Commit:     Adam Grare <agrare>
CommitDate: Mon May 8 19:13:38 2017 -0400

    Retrieve host storage devices host-by-host
    
    To reduce memory usage retrieve host storage devices host by host
    instead of in one call.  This was originally done to increase
    performance but the vast majority of the overhead was due to running
    garbage collection on every API call.  Now that we don't do this the
    performance increase of retrieving all hosts in one call is greatly
    reduced.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1448963

 .../providers/vmware/infra_manager/refresher.rb    |   17 +-
 .../miq_vim_inventory/hostSystemsStorageDevice.yml | 6613 --------------------
 .../vim_data/miq_vim_inventory/storageDevice.yml   | 5799 +++++++++++++++++
 3 files changed, 5810 insertions(+), 6619 deletions(-)
 delete mode 100644 spec/tools/vim_data/miq_vim_inventory/hostSystemsStorageDevice.yml
 create mode 100644 spec/tools/vim_data/miq_vim_inventory/storageDevice.yml
Comment 11 luke couzens 2017-11-03 11:05:56 UTC 
Verified in 5.9.0.4