Description of problem: Full refresh is broken with the below logs, and it seems kubeclient is trying to connect to older endpoint ``` [----] I, [2019-01-21T07:32:54.758192 #15170:5f4f4c] INFO -- : MIQ(MiqQueue.put) Message id: [1658], id: [], Zone: [default], Role: [automate], Server: [], MiqTask id: [], Ident: [generic], Target id: [], Inst ance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: [{:object_type=>"ManageIQ::Providers::Openshift::ContainerManager", :obj ect_id=>1, :attrs=>{:event_type=>"ems_auth_valid", "MiqEvent::miq_event"=>88, :miq_event_id=>88, "EventStream::event_stream"=>88, :event_stream_id=>88}, :instance_name=>"Event", :user_id=>1, :miq_group_id=>1, :t enant_id=>1, :automate_message=>nil}] [----] I, [2019-01-21T07:32:54.758396 #15170:5f4f4c] INFO -- : MIQ(MiqQueue#delivered) Message id: [1649], State: [ok], Delivered in [0.020613737] seconds /opt/rh/cfme-gemset/gems/fog-kubevirt-0.1.8/lib/fog/compute/kubevirt.rb:79:in `rescue in method_missing': HTTP status code 404, 404 Not Found for GET https://10.8.247.188:8443/apis/kubevirt.io/v1alpha2 (Fog::Kub evirt::Errors::ClientError) from /opt/rh/cfme-gemset/gems/fog-kubevirt-0.1.8/lib/fog/compute/kubevirt.rb:68:in `method_missing' from /opt/rh/cfme-gemset/gems/fog-kubevirt-0.1.8/lib/fog/compute/kubevirt.rb:239:in `watch_vms' from /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/infra_manager/connection.rb:138:in `watch_vms' from /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/infra_manager/refresh_worker/runner.rb:207:in `block in start_watches' from /var/www/miq/vmdb/app/models/ext_management_system.rb:413:in `with_provider_connection' from /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/infra_manager/refresh_worker/runner.rb:205:in `start_watches' from /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/infra_manager/refresh_worker/runner.rb:57:in `do_before_work_loop' from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:148:in `prepare' from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:126:in `start' from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker' from /var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork' from /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' from /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' from /var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork' from /var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner' from /var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start' from /var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker' from /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems' from /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers' from /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each' from /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers' from /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers' from /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' from /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' from /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' from /var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor' from /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' from /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block' from /var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor' from /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop' from /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' from /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block' from /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop' from /var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop' from /var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop' from /var/www/miq/vmdb/app/models/miq_server.rb:241:in `start' from /var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start' from /var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start' from /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' [----] I, [2019-01-21T07:32:55.667016 #8881:5f4f4c] INFO -- : EMS: [kubevirt], id: [1] Saving EMS Inventory... [----] I, [2019-01-21T07:32:56.521061 #8881:5f4f4c] INFO -- : EMS: [kubevirt], id: [1] Saving EMS Inventory...Complete [----] I, [2019-01-21T07:32:56.521241 #8881:5f4f4c] INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh_targets_for_ems) EMS: [kubevirt], id: [1] Refreshing target ManageIQ::Providers::Openshift::ContainerManager [kubevirt] id [1]...Complete ``` [----] I, [2019-01-21T07:32:54.706039 #15170:5f4f4c] INFO -- : MIQ(MiqQueue#deliver) Message id: [1648], Delivering... [----] E, [2019-01-21T07:32:54.722430 #8862:5f4f4c] ERROR -- : MIQ(ManageIQ::Providers::Kubevirt::InfraManager::RefreshWorker::Runner#full_refresh) Full refresh failed. [----] E, [2019-01-21T07:32:54.724818 #8862:5f4f4c] ERROR -- : [NoMethodError]: undefined method `kind' for []:Array Method:[block (2 levels) in <class:LogProxy>] [----] E, [2019-01-21T07:32:54.725001 #8862:5f4f4c] ERROR -- : /opt/rh/cfme-gemset/gems/fog-kubevirt-0.1.8/lib/fog/compute/kubevirt/models/vms.rb:21:in `all' /opt/rh/cfme-gemset/gems/fog-core-1.45.0/lib/fog/core/collection.rb:112:in `lazy_load' /opt/rh/cfme-gemset/gems/fog-core-1.45.0/lib/fog/core/collection.rb:16:in `each' /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/inventory/parser.rb:112:in `process_vms' /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/inventory/parser/full_refresh.rb:47:in `parse' /var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:37:in `block in parse' /var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:34:in `each' /var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:34:in `parse' /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/infra_manager/refresh_worker/runner.rb:98:in `full_refresh' /opt/rh/cfme-gemset/bundler/gems/cfme-providers-kubevirt-78df33dae029/app/models/manageiq/providers/kubevirt/infra_manager/refresh_worker/runner.rb:22:in `do_before_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:148:in `prepare' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:126:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork' /var/www/miq/vmdb/app/models/miq_worker.rb:390:in `start_runner' /var/www/miq/vmdb/app/models/miq_worker.rb:441:in `start' /var/www/miq/vmdb/app/models/miq_worker.rb:271:in `start_worker' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:74:in `start_worker_for_ems' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:52:in `block in sync_workers' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `each' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:51:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' /var/www/miq/vmdb/app/models/miq_server.rb:339:in `block in monitor' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:339:in `monitor' /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block (2 levels) in monitor_loop' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/cfme-gems-pending-71d6e5aabd0b/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:380:in `block in monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:in `loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:in `monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:241:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' [----] I, [2019-01-21T07:32:54.727313 #15170:5f4f4c] INFO -- : MIQ(MiqQueue.put) Message id: [1657], id: [], Zone: [default], Role: [automate], Server: [], MiqTask id: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: [{:object_type=>"ManageIQ::Providers::Openshift::ContainerManager", :object_id=>1, :attrs=>{:event_type=>"ems_auth_valid", "MiqEvent::miq_event"=>87, :miq_event_id=>87, "EventStream::event_stream"=>87, :event_stream_id=>87}, :instance_name=>"Event", :user_id=>1, :miq_group_id=>1, :tenant_id=>1, :automate_message=>nil}] Version-Release number of selected component (if applicable): Version 5.10.0.32.20190115185124_c957ada How reproducible: Steps to Reproduce: 1.Add CNV provider 2. 3. Actual results: throw Error about refresh with above logs Expected results: Refresh should work Additional info:
Once the PR is merged we need to release new version of fog-kubevirt and update the provider dependency.
In general this BZ is about usage of newer Kubevirt using new api version. With previous version it would work as expected.
Which Kubevirt version are you using. I installed v0.13.0 and I see v1alpha3 version. Whereas in your environment I see v1beta1.
I am using CNV 1.4 d/s + OCP 3.11
Fabian, what should be correct api version?
KubeVirt 0.13 which is in CNV 1.4 moved to v1alpha3. However, you should use K8s introspection (kubectl api-versions like) to detect what kubevirt api is available.
New commit detected on ManageIQ/manageiq-providers-kubevirt/master: https://github.com/ManageIQ/manageiq-providers-kubevirt/commit/991a45b8a0639181ad9572c70864999e935d652b commit 991a45b8a0639181ad9572c70864999e935d652b Author: Piotr Kliczewski <piotr.kliczewski> AuthorDate: Wed Jan 23 08:22:44 2019 -0500 Commit: Piotr Kliczewski <piotr.kliczewski> CommitDate: Wed Jan 23 08:22:44 2019 -0500 fog: update version We need to update fog-kubevirt version to contain api versio detection. Fixes: https://bugzilla.redhat.com/1667938 manageiq-providers-kubevirt.gemspec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)