Bug 1470021

Summary: [RFE] Implement Inventory Graph Refresh for OpenShift to improve collection performance
Product: Red Hat CloudForms Management Engine Reporter: Federico Simoncelli <fsimonce>
Component: ProvidersAssignee: Beni Paskin-Cherniavsky <cben>
Status: CLOSED CURRENTRELEASE QA Contact: juwatts
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: agrare, cpelland, dajohnso, epacific, gblomqui, jfrey, jhardy, lsmola, obarenbo, snaim
Target Milestone: GAKeywords: FutureFeature, TestOnly
Target Release: 5.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: container
Fixed In Version: 5.10.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1520488 (view as bug list) Environment:
Last Closed: 2018-06-21 20:56:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Container Management Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1520488    

Description Federico Simoncelli 2017-07-12 09:44:09 UTC 
Description of problem:
Implement Inventory Graph Refresh for OpenShift to improve collection performance.
Comment 2 Beni Paskin-Cherniavsky 2017-09-15 15:44:02 UTC 
Neglected updating this, it has been in development for months.
It's working (though needs more testing), with exception of label->tag mapping.
We about to turn it on by default (CF 4.6 / CFME 5.9 will keep both code paths, with a setting that will allow reverting to old code; CF 5.0 is expected to drop old containers refresh).

Also tracked on https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/27

To clarify, this will still be full refresh, not targetted, but using the InventoryObject/Collection (aka "graph refresh") infrastructure to dramatically reduce saving time and RAM usage.
Targetted refresh will be tracked separately (but this effort did bring us closer to implementing it).

Incomplete list of merged PRs (by many authors) that contributed to this so far:
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/30
https://github.com/ManageIQ/manageiq/pull/15298
https://github.com/ManageIQ/manageiq-providers-openshift/pull/18
https://github.com/ManageIQ/manageiq-providers-openshift/pull/19
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/46
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/52
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/73
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/74
https://github.com/ManageIQ/manageiq-providers-openshift/pull/30
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/57
https://github.com/ManageIQ/manageiq-providers-openshift/pull/34
https://github.com/ManageIQ/manageiq-providers-openshift/pull/39
https://github.com/ManageIQ/manageiq/pull/15251
https://github.com/ManageIQ/manageiq-schema/pull/18
https://github.com/ManageIQ/manageiq/pull/15247
https://github.com/ManageIQ/manageiq/pull/15576
https://github.com/ManageIQ/manageiq/pull/15603
https://github.com/ManageIQ/manageiq/pull/15627
https://github.com/ManageIQ/manageiq/pull/15698
https://github.com/ManageIQ/manageiq/pull/15761
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/88
https://github.com/ManageIQ/manageiq/pull/15699
https://github.com/ManageIQ/manageiq/pull/15679
https://github.com/ManageIQ/manageiq/pull/15678
https://github.com/ManageIQ/manageiq/pull/15447
https://github.com/ManageIQ/manageiq/pull/15648
https://github.com/ManageIQ/manageiq/pull/15687
https://github.com/ManageIQ/manageiq/pull/15804
https://github.com/ManageIQ/manageiq/pull/15814
(Not really sorted.  Unfairly omits more work in core ManageIQ I'm not aware of.  There are also significant non-code contributions, notably performance team doing measurements.)
Comment 3 CFME Bot 2017-09-15 16:14:06 UTC 
https://github.com/ManageIQ/manageiq-providers-openshift/pull/48
Comment 4 CFME Bot 2017-09-15 16:14:13 UTC 
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/112
Comment 5 CFME Bot 2017-09-18 12:59:22 UTC 
New commit detected on ManageIQ/manageiq-providers-openshift/master:
https://github.com/ManageIQ/manageiq-providers-openshift/commit/0c117cf4de1c3d7acbbd20b7d947e7dc699f4872

commit 0c117cf4de1c3d7acbbd20b7d947e7dc699f4872
Author:     Beni Cherniavsky-Paskin <cben>
AuthorDate: Fri Sep 15 12:06:15 2017 -0400
Commit:     Beni Cherniavsky-Paskin <cben>
CommitDate: Fri Sep 15 12:06:15 2017 -0400

    Enable graph refresh (batch strategy) by default
    
    There are some loose ends (notably tag mapping missing)
    but we want to get more testing from developers using it.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1470021

 config/settings.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Comment 6 CFME Bot 2017-09-18 12:59:27 UTC 
New commit detected on ManageIQ/manageiq-providers-kubernetes/master:
https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/8e3c4889642cfd412b5bfabfdec70d08d943fd47

commit 8e3c4889642cfd412b5bfabfdec70d08d943fd47
Author:     Beni Cherniavsky-Paskin <cben>
AuthorDate: Fri Sep 15 12:03:54 2017 -0400
Commit:     Beni Cherniavsky-Paskin <cben>
CommitDate: Fri Sep 15 12:03:54 2017 -0400

    Enable graph refresh (batch strategy) by default
    
    There are some loose ends (notably tag mapping missing)
    but we want to get more testing from developers using it.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1470021

 config/settings.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Comment 7 Beni Paskin-Cherniavsky 2017-10-19 10:51:22 UTC 
Ladislav fixed bug with save_unused_images: false mode deleting existing child custom_attributes records:
https://github.com/ManageIQ/manageiq/pull/15903
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/134
https://github.com/ManageIQ/manageiq-providers-openshift/pull/46

working on tag mapping (https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/126)
Comment 8 CFME Bot 2017-11-13 18:33:39 UTC 
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/162
Comment 9 CFME Bot 2017-12-03 08:44:23 UTC 
New commit detected on ManageIQ/manageiq-providers-kubernetes/master:
https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/f793ed46a69867550ee3d3bd780ebe22ebf5da1c

commit f793ed46a69867550ee3d3bd780ebe22ebf5da1c
Author:     Beni Cherniavsky-Paskin <cben>
AuthorDate: Tue Oct 17 16:37:08 2017 +0300
Commit:     Beni Cherniavsky-Paskin <cben>
CommitDate: Fri Dec 1 07:33:50 2017 +0200

    Tag mapping in graph refresh
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1470021

 .../container_manager/inventory_collections.rb     | 37 +++++++++++++++++++++
 .../kubernetes/container_manager/refresh_parser.rb | 38 +++++++++++++++-------
 .../kubernetes/container_manager/refresher_spec.rb | 28 +++++++---------
 3 files changed, 75 insertions(+), 28 deletions(-)
Comment 10 Beni Paskin-Cherniavsky 2017-12-04 10:11:51 UTC 
Moving to POST, tag mapping finally works in graph refresh :-)

Merged PRs related to tag mapping (more detailed summaries in https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/126):

https://github.com/ManageIQ/manageiq/pull/16098
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/130
https://github.com/ManageIQ/manageiq-providers-amazon/pull/316
https://github.com/ManageIQ/manageiq/pull/16259
https://github.com/ManageIQ/manageiq/pull/16453
https://github.com/ManageIQ/manageiq/pull/16449
https://github.com/ManageIQ/manageiq/pull/16499
https://github.com/ManageIQ/manageiq/pull/16501
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/170
https://github.com/ManageIQ/manageiq-ui-classic/pull/2788

merged to master, still needs backporting to 5.9:

https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/162
https://github.com/ManageIQ/manageiq-providers-openshift/pull/64

unmerged cleanup, does not affect behavior:

https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/180