Description of problem: Upgrade from CNV 2.3 (production) to 2.4 (brew) - SSP oeprator fails to upgrade node-labeller and template-validator. Version-Release number of selected component (if applicable): 2.3.0 -> 2.4.0 How reproducible: 100% (2 out of 2 upgrades) Steps to Reproduce: 1. Install 2.3.0 from production 2. Create a new operator source pointing to rh-osbs-operators 3. Update the channel to 2.4 4. Approve the install plan Actual results: virt-template-validator and kubevirt-node-labeller pods are not upgraded Expected results: All pods should be upgraded successfully. Additional info: $ oc get crd |grep -i ssp kubevirtcommontemplatesbundles.ssp.kubevirt.io 2020-06-04T10:26:17Z kubevirtmetricsaggregations.ssp.kubevirt.io 2020-06-04T10:26:17Z kubevirtnodelabellerbundles.ssp.kubevirt.io 2020-06-04T10:26:17Z kubevirttemplatevalidators.ssp.kubevirt.io 2020-06-04T10:26:17Z $ oc logs -n openshift-cnv kubevirt-ssp-operator-58cd45bf85-7f2b9 {"level":"info","ts":1591281003.1349926,"logger":"cmd","msg":"Go Version: go1.13.4"} {"level":"info","ts":1591281003.135037,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1591281003.1350417,"logger":"cmd","msg":"Version of operator-sdk: v0.12.0+git"} {"level":"info","ts":1591281003.135061,"logger":"cmd","msg":"Watching namespace.","Namespace":""} {"level":"info","ts":1591281005.9466867,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"} {"level":"info","ts":1591281005.9476843,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTCOMMONTEMPLATESBUNDLE_SSP_KUBEVIRT_IO":1} {"level":"info","ts":1591281005.947707,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTCOMMONTEMPLATESBUNDLE_SSP_KUBEVIRT_IO":2} {"level":"info","ts":1591281005.947724,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTTEMPLATEVALIDATOR_SSP_KUBEVIRT_IO":1} {"level":"info","ts":1591281005.9477293,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTTEMPLATEVALIDATOR_SSP_KUBEVIRT_IO":2} {"level":"info","ts":1591281005.9477417,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTNODELABELLERBUNDLE_SSP_KUBEVIRT_IO":1} {"level":"info","ts":1591281005.947747,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTNODELABELLERBUNDLE_SSP_KUBEVIRT_IO":2} {"level":"info","ts":1591281005.9477563,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTMETRICSAGGREGATION_SSP_KUBEVIRT_IO":1} {"level":"info","ts":1591281005.9477618,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTMETRICSAGGREGATION_SSP_KUBEVIRT_IO":2} {"level":"info","ts":1591281005.9480221,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtCommonTemplatesBundle"} {"level":"info","ts":1591281005.9483616,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirtcommontemplatesbundle-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtCommonTemplatesBundle"} {"level":"info","ts":1591281005.9485798,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtTemplateValidator"} {"level":"info","ts":1591281005.9486978,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirttemplatevalidator-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtTemplateValidator"} {"level":"info","ts":1591281005.9488313,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtNodeLabellerBundle"} {"level":"info","ts":1591281005.9489481,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirtnodelabellerbundle-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle"} {"level":"info","ts":1591281005.9490542,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtMetricsAggregation"} {"level":"info","ts":1591281005.9491465,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirtmetricsaggregation-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtMetricsAggregation"} {"level":"info","ts":1591281005.9492555,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1591281008.7723699,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1591281008.7909398,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1591281022.878182,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"kubevirt-ssp-operator-metrics","Service.Namespace":"openshift-cnv"} {"level":"info","ts":1591281022.879933,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} {"level":"info","ts":1591281022.880182,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1591281022.9803753,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirtcommontemplatesbundle-controller"} {"level":"info","ts":1591281022.9804351,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirtnodelabellerbundle-controller"} {"level":"info","ts":1591281022.9804604,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirttemplatevalidator-controller"} {"level":"info","ts":1591281022.9804688,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirtmetricsaggregation-controller"} {"level":"info","ts":1591281023.0806372,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirtcommontemplatesbundle-controller","worker count":1} {"level":"info","ts":1591281023.080771,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirtnodelabellerbundle-controller","worker count":1} {"level":"info","ts":1591281023.0811253,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirttemplatevalidator-controller","worker count":1} {"level":"info","ts":1591281023.081312,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirtmetricsaggregation-controller","worker count":1} {"level":"info","ts":1591281025.108222,"logger":"logging_event_handler","msg":"[playbook task]","name":"common-templates-kubevirt-hyperconverged","namespace":"openshift","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtCommonTemplatesBundle","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"Gathering Facts"} {"level":"info","ts":1591281025.2750564,"logger":"logging_event_handler","msg":"[playbook task]","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"2775422040480279449","EventData.Name":"Gathering Facts"} {"level":"info","ts":1591281025.2882504,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"894385949183117216","EventData.Name":"Gathering Facts"} {"level":"error","ts":1591281027.020319,"logger":"logging_event_handler","msg":"","name":"metrics-aggregation-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtMetricsAggregation","event_type":"runner_on_failed","job":"4751997750760398084","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281027.2420487,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: metricsaggregation.yaml **********************************************\n\u001b[0;34m1 plays in /opt/ansible/metricsaggregation.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtMetricsAggregation by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : ok=0 changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"4751997750760398084","name":"metrics-aggregation-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"} {"level":"error","ts":1591281030.548158,"logger":"logging_event_handler","msg":"","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"2775422040480279449","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281030.7633631,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"2775422040480279449","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"} {"level":"error","ts":1591281030.874135,"logger":"logging_event_handler","msg":"","name":"common-templates-kubevirt-hyperconverged","namespace":"openshift","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtCommonTemplatesBundle","event_type":"runner_on_failed","job":"7504504064263669287","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281030.887357,"logger":"logging_event_handler","msg":"","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"runner_on_failed","job":"894385949183117216","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281031.1060274,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: commontemplatesbundle.yaml *******************************************\n\u001b[0;34m1 plays in /opt/ansible/commontemplatesbundle.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/commontemplatesbundle.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtCommonTemplatesBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"7504504064263669287","name":"common-templates-kubevirt-hyperconverged","namespace":"openshift","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"} {"level":"error","ts":1591281031.1152005,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: templatevalidator.yaml ***********************************************\n\u001b[0;34m1 plays in /opt/ansible/templatevalidator.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[1;30mtask path: /opt/ansible/templatevalidator.yaml:1\u001b[0m\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/templatevalidator.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtTemplateValidator by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"894385949183117216","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"} {"level":"info","ts":1591281032.64987,"logger":"logging_event_handler","msg":"[playbook task]","name":"kubevirt-node-labeller-bundle","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"1976235410884491574","EventData.Name":"Gathering Facts"} {"level":"error","ts":1591281037.57158,"logger":"logging_event_handler","msg":"","name":"kubevirt-node-labeller-bundle","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"1976235410884491574","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281037.7539446,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"1976235410884491574","name":"kubevirt-node-labeller-bundle","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"} {"level":"info","ts":1591281672.1888278,"logger":"logging_event_handler","msg":"[playbook task]","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"3328451335138149956","EventData.Name":"Gathering Facts"} {"level":"error","ts":1591281677.012553,"logger":"logging_event_handler","msg":"","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"3328451335138149956","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281677.191724,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"3328451335138149956","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"} {"level":"info","ts":1591281744.8202064,"logger":"logging_event_handler","msg":"[playbook task]","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"5263531936693774911","EventData.Name":"Gathering Facts"} {"level":"error","ts":1591281749.7325191,"logger":"logging_event_handler","msg":"","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"5263531936693774911","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1591281749.9451694,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 rescued=0 ignored=0 \r\n\n","job":"5263531936693774911","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
Requesting blocker+ flag, as upgrade to 2.4 must succeed.
A PR is open and can be followed here: https://github.com/MarSik/kubevirt-ssp-operator/pull/186
SSP PR is merged, HCO PR can be found here: https://github.com/kubevirt/hyperconverged-cluster-operator/pull/634
After the HCO team finishes their testing, both HCO and SSP operator will create d/s builds, as they are both required for the complete solution.
Verified: OCP 4.5.0-rc., upgrade from 2.3.0 - > 2.4.0 (SSP v2.4.0-66) ends successfully. Both virt-template-validator and kubevirt-node-labeller pods are upgraded.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194