Bug 1470021 - [RFE] Implement Inventory Graph Refresh for OpenShift to improve collection performance
[RFE] Implement Inventory Graph Refresh for OpenShift to improve collection p...
Status: POST
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
unspecified
Unspecified Unspecified
high Severity high
: GA
: 6.0.0
Assigned To: Beni Paskin-Cherniavsky
juwatts
container
: FutureFeature, TestOnly
Depends On:
Blocks: 1520488
  Show dependency treegraph
 
Reported: 2017-07-12 05:44 EDT by Federico Simoncelli
Modified: 2018-01-05 18:27 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1520488 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Container Management


Attachments (Terms of Use)

  None (edit)
Description Federico Simoncelli 2017-07-12 05:44:09 EDT
Description of problem:
Implement Inventory Graph Refresh for OpenShift to improve collection performance.
Comment 2 Beni Paskin-Cherniavsky 2017-09-15 11:44:02 EDT
Neglected updating this, it has been in development for months.
It's working (though needs more testing), with exception of label->tag mapping.
We about to turn it on by default (CF 4.6 / CFME 5.9 will keep both code paths, with a setting that will allow reverting to old code; CF 5.0 is expected to drop old containers refresh).

Also tracked on https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/27

To clarify, this will still be full refresh, not targetted, but using the InventoryObject/Collection (aka "graph refresh") infrastructure to dramatically reduce saving time and RAM usage.
Targetted refresh will be tracked separately (but this effort did bring us closer to implementing it).

Incomplete list of merged PRs (by many authors) that contributed to this so far:
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/30
https://github.com/ManageIQ/manageiq/pull/15298
https://github.com/ManageIQ/manageiq-providers-openshift/pull/18
https://github.com/ManageIQ/manageiq-providers-openshift/pull/19
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/46
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/52
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/73
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/74
https://github.com/ManageIQ/manageiq-providers-openshift/pull/30
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/57
https://github.com/ManageIQ/manageiq-providers-openshift/pull/34
https://github.com/ManageIQ/manageiq-providers-openshift/pull/39
https://github.com/ManageIQ/manageiq/pull/15251
https://github.com/ManageIQ/manageiq-schema/pull/18
https://github.com/ManageIQ/manageiq/pull/15247
https://github.com/ManageIQ/manageiq/pull/15576
https://github.com/ManageIQ/manageiq/pull/15603
https://github.com/ManageIQ/manageiq/pull/15627
https://github.com/ManageIQ/manageiq/pull/15698
https://github.com/ManageIQ/manageiq/pull/15761
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/88
https://github.com/ManageIQ/manageiq/pull/15699
https://github.com/ManageIQ/manageiq/pull/15679
https://github.com/ManageIQ/manageiq/pull/15678
https://github.com/ManageIQ/manageiq/pull/15447
https://github.com/ManageIQ/manageiq/pull/15648
https://github.com/ManageIQ/manageiq/pull/15687
https://github.com/ManageIQ/manageiq/pull/15804
https://github.com/ManageIQ/manageiq/pull/15814
(Not really sorted.  Unfairly omits more work in core ManageIQ I'm not aware of.  There are also significant non-code contributions, notably performance team doing measurements.)
Comment 5 CFME Bot 2017-09-18 08:59:22 EDT
New commit detected on ManageIQ/manageiq-providers-openshift/master:
https://github.com/ManageIQ/manageiq-providers-openshift/commit/0c117cf4de1c3d7acbbd20b7d947e7dc699f4872

commit 0c117cf4de1c3d7acbbd20b7d947e7dc699f4872
Author:     Beni Cherniavsky-Paskin <cben@redhat.com>
AuthorDate: Fri Sep 15 12:06:15 2017 -0400
Commit:     Beni Cherniavsky-Paskin <cben@redhat.com>
CommitDate: Fri Sep 15 12:06:15 2017 -0400

    Enable graph refresh (batch strategy) by default
    
    There are some loose ends (notably tag mapping missing)
    but we want to get more testing from developers using it.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1470021

 config/settings.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Comment 6 CFME Bot 2017-09-18 08:59:27 EDT
New commit detected on ManageIQ/manageiq-providers-kubernetes/master:
https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/8e3c4889642cfd412b5bfabfdec70d08d943fd47

commit 8e3c4889642cfd412b5bfabfdec70d08d943fd47
Author:     Beni Cherniavsky-Paskin <cben@redhat.com>
AuthorDate: Fri Sep 15 12:03:54 2017 -0400
Commit:     Beni Cherniavsky-Paskin <cben@redhat.com>
CommitDate: Fri Sep 15 12:03:54 2017 -0400

    Enable graph refresh (batch strategy) by default
    
    There are some loose ends (notably tag mapping missing)
    but we want to get more testing from developers using it.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1470021

 config/settings.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Comment 7 Beni Paskin-Cherniavsky 2017-10-19 06:51:22 EDT
Ladislav fixed bug with save_unused_images: false mode deleting existing child custom_attributes records:
https://github.com/ManageIQ/manageiq/pull/15903
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/134
https://github.com/ManageIQ/manageiq-providers-openshift/pull/46

working on tag mapping (https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/126)
Comment 9 CFME Bot 2017-12-03 03:44:23 EST
New commit detected on ManageIQ/manageiq-providers-kubernetes/master:
https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/f793ed46a69867550ee3d3bd780ebe22ebf5da1c

commit f793ed46a69867550ee3d3bd780ebe22ebf5da1c
Author:     Beni Cherniavsky-Paskin <cben@redhat.com>
AuthorDate: Tue Oct 17 16:37:08 2017 +0300
Commit:     Beni Cherniavsky-Paskin <cben@redhat.com>
CommitDate: Fri Dec 1 07:33:50 2017 +0200

    Tag mapping in graph refresh
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1470021

 .../container_manager/inventory_collections.rb     | 37 +++++++++++++++++++++
 .../kubernetes/container_manager/refresh_parser.rb | 38 +++++++++++++++-------
 .../kubernetes/container_manager/refresher_spec.rb | 28 +++++++---------
 3 files changed, 75 insertions(+), 28 deletions(-)

Note You need to log in before you can comment on or make changes to this bug.