Description of problem: Implement Inventory Graph Refresh for OpenShift to improve collection performance.
Neglected updating this, it has been in development for months. It's working (though needs more testing), with exception of label->tag mapping. We about to turn it on by default (CF 4.6 / CFME 5.9 will keep both code paths, with a setting that will allow reverting to old code; CF 5.0 is expected to drop old containers refresh). Also tracked on https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/27 To clarify, this will still be full refresh, not targetted, but using the InventoryObject/Collection (aka "graph refresh") infrastructure to dramatically reduce saving time and RAM usage. Targetted refresh will be tracked separately (but this effort did bring us closer to implementing it). Incomplete list of merged PRs (by many authors) that contributed to this so far: https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/30 https://github.com/ManageIQ/manageiq/pull/15298 https://github.com/ManageIQ/manageiq-providers-openshift/pull/18 https://github.com/ManageIQ/manageiq-providers-openshift/pull/19 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/46 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/52 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/73 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/74 https://github.com/ManageIQ/manageiq-providers-openshift/pull/30 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/57 https://github.com/ManageIQ/manageiq-providers-openshift/pull/34 https://github.com/ManageIQ/manageiq-providers-openshift/pull/39 https://github.com/ManageIQ/manageiq/pull/15251 https://github.com/ManageIQ/manageiq-schema/pull/18 https://github.com/ManageIQ/manageiq/pull/15247 https://github.com/ManageIQ/manageiq/pull/15576 https://github.com/ManageIQ/manageiq/pull/15603 https://github.com/ManageIQ/manageiq/pull/15627 https://github.com/ManageIQ/manageiq/pull/15698 https://github.com/ManageIQ/manageiq/pull/15761 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/88 https://github.com/ManageIQ/manageiq/pull/15699 https://github.com/ManageIQ/manageiq/pull/15679 https://github.com/ManageIQ/manageiq/pull/15678 https://github.com/ManageIQ/manageiq/pull/15447 https://github.com/ManageIQ/manageiq/pull/15648 https://github.com/ManageIQ/manageiq/pull/15687 https://github.com/ManageIQ/manageiq/pull/15804 https://github.com/ManageIQ/manageiq/pull/15814 (Not really sorted. Unfairly omits more work in core ManageIQ I'm not aware of. There are also significant non-code contributions, notably performance team doing measurements.)
https://github.com/ManageIQ/manageiq-providers-openshift/pull/48
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/112
New commit detected on ManageIQ/manageiq-providers-openshift/master: https://github.com/ManageIQ/manageiq-providers-openshift/commit/0c117cf4de1c3d7acbbd20b7d947e7dc699f4872 commit 0c117cf4de1c3d7acbbd20b7d947e7dc699f4872 Author: Beni Cherniavsky-Paskin <cben> AuthorDate: Fri Sep 15 12:06:15 2017 -0400 Commit: Beni Cherniavsky-Paskin <cben> CommitDate: Fri Sep 15 12:06:15 2017 -0400 Enable graph refresh (batch strategy) by default There are some loose ends (notably tag mapping missing) but we want to get more testing from developers using it. https://bugzilla.redhat.com/show_bug.cgi?id=1470021 config/settings.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
New commit detected on ManageIQ/manageiq-providers-kubernetes/master: https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/8e3c4889642cfd412b5bfabfdec70d08d943fd47 commit 8e3c4889642cfd412b5bfabfdec70d08d943fd47 Author: Beni Cherniavsky-Paskin <cben> AuthorDate: Fri Sep 15 12:03:54 2017 -0400 Commit: Beni Cherniavsky-Paskin <cben> CommitDate: Fri Sep 15 12:03:54 2017 -0400 Enable graph refresh (batch strategy) by default There are some loose ends (notably tag mapping missing) but we want to get more testing from developers using it. https://bugzilla.redhat.com/show_bug.cgi?id=1470021 config/settings.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Ladislav fixed bug with save_unused_images: false mode deleting existing child custom_attributes records: https://github.com/ManageIQ/manageiq/pull/15903 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/134 https://github.com/ManageIQ/manageiq-providers-openshift/pull/46 working on tag mapping (https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/126)
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/162
New commit detected on ManageIQ/manageiq-providers-kubernetes/master: https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/f793ed46a69867550ee3d3bd780ebe22ebf5da1c commit f793ed46a69867550ee3d3bd780ebe22ebf5da1c Author: Beni Cherniavsky-Paskin <cben> AuthorDate: Tue Oct 17 16:37:08 2017 +0300 Commit: Beni Cherniavsky-Paskin <cben> CommitDate: Fri Dec 1 07:33:50 2017 +0200 Tag mapping in graph refresh https://bugzilla.redhat.com/show_bug.cgi?id=1470021 .../container_manager/inventory_collections.rb | 37 +++++++++++++++++++++ .../kubernetes/container_manager/refresh_parser.rb | 38 +++++++++++++++------- .../kubernetes/container_manager/refresher_spec.rb | 28 +++++++--------- 3 files changed, 75 insertions(+), 28 deletions(-)
Moving to POST, tag mapping finally works in graph refresh :-) Merged PRs related to tag mapping (more detailed summaries in https://github.com/ManageIQ/manageiq-providers-kubernetes/issues/126): https://github.com/ManageIQ/manageiq/pull/16098 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/130 https://github.com/ManageIQ/manageiq-providers-amazon/pull/316 https://github.com/ManageIQ/manageiq/pull/16259 https://github.com/ManageIQ/manageiq/pull/16453 https://github.com/ManageIQ/manageiq/pull/16449 https://github.com/ManageIQ/manageiq/pull/16499 https://github.com/ManageIQ/manageiq/pull/16501 https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/170 https://github.com/ManageIQ/manageiq-ui-classic/pull/2788 merged to master, still needs backporting to 5.9: https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/162 https://github.com/ManageIQ/manageiq-providers-openshift/pull/64 unmerged cleanup, does not affect behavior: https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/180