Description of problem: Adding Openshift Container Platform v3.6 to CloudForms 4.6 it discover and refresh most of the objects but it fail the refresh when its trying to collect the container services [----] D, [2018-03-16T16:27:33.893726 #30428:c1b13c] DEBUG -- : MIQ(ManagerRefresh::SaveCollection::Saver::Default#save!) Processing InventoryCollection:<ContainerService>, blacklist: [namespace] of size 44... [----] E, [2018-03-16T16:27:33.905039 #30428:c1b13c] ERROR -- : MIQ(ManagerRefresh::SaveCollection::Saver::Default#save!) Error when saving InventoryCollection:<ContainerService>, blacklist: [namespace] with strategy: , saver_strategy: default, targeted: false. Message: undefined method `each' for nil:NilClass container_services; // InventoryCollection:<ContainerService>, blacklist: [namespace] container_service_port_configs; // InventoryCollection:<ContainerServicePortConfig> [----] D, [2018-03-16T16:32:03.813700 #30428:c1b13c] DEBUG -- : MIQ(ManagerRefresh::SaveCollection::TopologicalSort.save_inventory_object_inventory) Saving collection InventoryCollection:<ContainerService>, blacklist: [namespace] of size 44 to the database, for the manager: 'Non-Prod Openshift Container Platform'... [----] D, [2018-03-16T16:32:03.817998 #30428:c1b13c] DEBUG -- : MIQ(ManagerRefresh::SaveCollection::Saver::Default#save!) Processing InventoryCollection:<ContainerService>, blacklist: [namespace] of size 44... [----] E, [2018-03-16T16:32:03.829032 #30428:c1b13c] ERROR -- : MIQ(ManagerRefresh::SaveCollection::Saver::Default#save!) Error when saving InventoryCollection:<ContainerService>, blacklist: [namespace] with strategy: , saver_strategy: default, targeted: false. Message: undefined method `each' for nil:NilClass container_services; // InventoryCollection:<ContainerService>, blacklist: [namespace] container_service_port_configs; // InventoryCollection:<ContainerServicePortConfig> Version-Release number of selected component (if applicable): 5.9.1.0.20180314203929_67fd99d How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The issue is ContainerService.create!(:container_groups => nil) in https://github.com/Ladas/manageiq-providers-kubernetes/blob/f9bb30cde987500d482bd5507613b823bc3d3f5b/app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb#L542 so it needs to be: h[:container_groups] = cgs_by_namespace_and_name.fetch_path(h[:namespace], h[:name]) || []
Not sure if it was clear by the previous comment, but this is a Rails thing, example: ContainerService.create!(:container_groups => nil) will fail (with the same error as we see in log) ContainerService.create!(:container_groups => []) will pass
Thanks! Working on PR.
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/242
https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/242 (fix + test) https://github.com/ManageIQ/manageiq-providers-openshift/pull/92 (test) The exact condition that triggers this is Service without matching Endpoints. - This can happen for services without a selector. Such services are legitimate, but not really useful until one manually creates an Endpoints object: https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors Confirmed that creating an empty Endpoints would avoid the crash. - The PRs possibly also fix a race condition — refresh does separate requests for services and endpoints, so could in theory see one without the other. Bug present both in graph and classic refresh, so might be relevant to 5.8 or older too ? (untested)
New commits detected on ManageIQ/manageiq-providers-kubernetes/master: https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/7350379389a70ad606380cbdb8ac6dc5e6a4d1d6 commit 7350379389a70ad606380cbdb8ac6dc5e6a4d1d6 Author: Beni Cherniavsky-Paskin <cben> AuthorDate: Wed Mar 21 13:27:35 2018 -0400 Commit: Beni Cherniavsky-Paskin <cben> CommitDate: Wed Mar 21 13:27:35 2018 -0400 [classic refresh] Fix refresh crash on Service without Endpoints Avoids "undefined method `map' for nil:NilClass" https://bugzilla.redhat.com/show_bug.cgi?id=1558209 app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb | 1 + 1 file changed, 1 insertion(+) https://github.com/ManageIQ/manageiq-providers-kubernetes/commit/b5af5e078cd0fc3ca1c4acfe40a5594ac00e0024 commit b5af5e078cd0fc3ca1c4acfe40a5594ac00e0024 Author: Beni Cherniavsky-Paskin <cben> AuthorDate: Wed Mar 21 13:40:42 2018 -0400 Commit: Beni Cherniavsky-Paskin <cben> CommitDate: Wed Mar 21 13:40:42 2018 -0400 [graph refresh] Fix refresh crash on Service without Endpoints Avoids "undefined method `each' for nil:NilClass" https://bugzilla.redhat.com/show_bug.cgi?id=1558209 app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
*** Bug 1568040 has been marked as a duplicate of this bug. ***
Verified on: 5.10.0.20.20181016163900_fe677b4 Verification steps: 1) On the Openshift provider, created the following service json file: # cat service_no_endpoint.json { "kind": "Service", "apiVersion": "v1", "metadata": { "name": "hello-service" }, "spec": { "ports": [ { "protocol": "TCP", "port": 8888, "targetPort": 8080 } ] } } 2) Added the service by running oc create -f service_no_endpoint.json # oc get service NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-service <IP_Removed> <none> 8888/TCP 5s # 3) On the CFME appliance, triggered a manual refresh [----] I, [2018-10-24T10:49:52.423644 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) Refreshing all targets... [----] I, [2018-10-24T10:49:52.423786 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) EMS: [ocp-36-hawk], id: [1] Refreshing targets for EMS... [----] I, [2018-10-24T10:49:52.423902 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) EMS: [ocp-36-hawk], id: [1] ManageIQ::Providers::Openshift::ContainerManager [ocp-36-hawk] id [1] [----] I, [2018-10-24T10:49:52.424085 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager#with_provider_connection) Connecting through ManageIQ::Providers::Openshift::ContainerManager: [ocp-36-hawk] [----] I, [2018-10-24T10:49:52.640552 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager#with_provider_connection) Connecting through ManageIQ::Providers::Openshift::ContainerManager: [ocp-36-hawk] [----] I, [2018-10-24T10:49:53.959485 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh_targets_for_ems) EMS: [ocp-36-hawk], id: [1] Refreshing target ManageIQ::Providers::Openshift::ContainerManager [ocp-36-hawk] id [1]... [----] I, [2018-10-24T10:49:56.399006 #21231:b0af7c] INFO -- : EMS: [ocp-36-hawk], id: [1] Saving EMS Inventory... [----] I, [2018-10-24T10:49:58.216387 #12239:b0af7c] INFO -- : MIQ(MiqScheduleWorker::Runner#do_work) Number of scheduled items to be processed: 1. [----] I, [2018-10-24T10:49:59.603220 #21231:b0af7c] INFO -- : EMS: [ocp-36-hawk], id: [1] Saving EMS Inventory...Complete [----] I, [2018-10-24T10:49:59.603435 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh_targets_for_ems) EMS: [ocp-36-hawk], id: [1] Refreshing target ManageIQ::Providers::Openshift::ContainerManager [ocp-36-hawk] id [1]...Complete [----] I, [2018-10-24T10:49:59.603571 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh_targets_for_ems) EMS: [ocp-36-hawk], id: [1] ManagerRefresh Post Processing ManageIQ::Providers::Openshift::ContainerManager [ocp-36-hawk] id [1]... [----] I, [2018-10-24T10:49:59.603706 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh_targets_for_ems) EMS: [ocp-36-hawk], id: [1] ManagerRefresh Post Processing ManageIQ::Providers::Openshift::ContainerManager [ocp-36-hawk] id [1]...Complete [----] I, [2018-10-24T10:49:59.603798 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) EMS: [ocp-36-hawk], id: [1] Refreshing targets for EMS...Complete - Timings {:collect_inventory_for_targets=>1.5353152751922607, :parse_targeted_inventory=>2.146233558654785, :save_inventory=>3.497368097305298, :manager_refresh_post_processing=>1.3589859008789062e-05, :ems_refresh=>7.179760217666626} [----] I, [2018-10-24T10:49:59.617810 #21231:b0af7c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) Refreshing all targets...Complete