Bug 1844105 - [SSP operator] Upgrade from 2.3.0 to 2.4.0- SSP operator fails to upgrade node labeller and template validator
Summary: [SSP operator] Upgrade from 2.3.0 to 2.4.0- SSP operator fails to upgrade nod...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: SSP
Version: 2.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 2.4.0
Assignee: Omer Yahud
QA Contact: Israel Pinto
URL:
Whiteboard:
Depends On:
Blocks: 1832291
TreeView+ depends on / blocked
 
Reported: 2020-06-04 16:01 UTC by Ruth Netser
Modified: 2020-07-28 19:10 UTC (History)
6 users (show)

Fixed In Version: kubevirt-ssp-operator-container-v2.4.0-61, hco-bundle-registry-container-v2.3.0-435
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-28 19:10:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github MarSik kubevirt-ssp-operator pull 186 0 None closed Implement upgrade path for SSP operator 2020-07-22 10:32:46 UTC
Github kubevirt hyperconverged-cluster-operator pull 635 0 None closed [release-2.4] Improve the upgrade detection 2020-07-22 10:32:45 UTC
Github kubevirt hyperconverged-cluster-operator pull 657 0 None closed Bump SSP to Version v1.0.38 2020-07-22 10:32:45 UTC
Red Hat Product Errata RHSA-2020:3194 0 None None None 2020-07-28 19:10:43 UTC

Description Ruth Netser 2020-06-04 16:01:59 UTC
Description of problem:
Upgrade from CNV 2.3 (production) to 2.4 (brew) - SSP oeprator fails to upgrade node-labeller and template-validator.

Version-Release number of selected component (if applicable):
2.3.0 -> 2.4.0

How reproducible:
100% (2 out of 2 upgrades)


Steps to Reproduce:
1. Install 2.3.0 from production
2. Create a new operator source pointing to rh-osbs-operators
3. Update the channel to 2.4
4. Approve the install plan

Actual results:
virt-template-validator and kubevirt-node-labeller pods are not upgraded

Expected results:
All pods should be upgraded successfully.


Additional info:

$ oc get crd |grep -i ssp
kubevirtcommontemplatesbundles.ssp.kubevirt.io                   2020-06-04T10:26:17Z
kubevirtmetricsaggregations.ssp.kubevirt.io                      2020-06-04T10:26:17Z
kubevirtnodelabellerbundles.ssp.kubevirt.io                      2020-06-04T10:26:17Z
kubevirttemplatevalidators.ssp.kubevirt.io                       2020-06-04T10:26:17Z


$ oc logs -n openshift-cnv  kubevirt-ssp-operator-58cd45bf85-7f2b9
{"level":"info","ts":1591281003.1349926,"logger":"cmd","msg":"Go Version: go1.13.4"}
{"level":"info","ts":1591281003.135037,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1591281003.1350417,"logger":"cmd","msg":"Version of operator-sdk: v0.12.0+git"}
{"level":"info","ts":1591281003.135061,"logger":"cmd","msg":"Watching namespace.","Namespace":""}
{"level":"info","ts":1591281005.9466867,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1591281005.9476843,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTCOMMONTEMPLATESBUNDLE_SSP_KUBEVIRT_IO":1}
{"level":"info","ts":1591281005.947707,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTCOMMONTEMPLATESBUNDLE_SSP_KUBEVIRT_IO":2}
{"level":"info","ts":1591281005.947724,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTTEMPLATEVALIDATOR_SSP_KUBEVIRT_IO":1}
{"level":"info","ts":1591281005.9477293,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTTEMPLATEVALIDATOR_SSP_KUBEVIRT_IO":2}
{"level":"info","ts":1591281005.9477417,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTNODELABELLERBUNDLE_SSP_KUBEVIRT_IO":1}
{"level":"info","ts":1591281005.947747,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTNODELABELLERBUNDLE_SSP_KUBEVIRT_IO":2}
{"level":"info","ts":1591281005.9477563,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","WORKER_KUBEVIRTMETRICSAGGREGATION_SSP_KUBEVIRT_IO":1}
{"level":"info","ts":1591281005.9477618,"logger":"watches","msg":"Failed to parse %v from environment. Using default %v","ANSIBLE_VERBOSITY_KUBEVIRTMETRICSAGGREGATION_SSP_KUBEVIRT_IO":2}
{"level":"info","ts":1591281005.9480221,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtCommonTemplatesBundle"}
{"level":"info","ts":1591281005.9483616,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirtcommontemplatesbundle-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtCommonTemplatesBundle"}
{"level":"info","ts":1591281005.9485798,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtTemplateValidator"}
{"level":"info","ts":1591281005.9486978,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirttemplatevalidator-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtTemplateValidator"}
{"level":"info","ts":1591281005.9488313,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtNodeLabellerBundle"}
{"level":"info","ts":1591281005.9489481,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirtnodelabellerbundle-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle"}
{"level":"info","ts":1591281005.9490542,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"ssp.kubevirt.io","Options.Version":"v1","Options.Kind":"KubevirtMetricsAggregation"}
{"level":"info","ts":1591281005.9491465,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kubevirtmetricsaggregation-controller","source":"kind source: ssp.kubevirt.io/v1, Kind=KubevirtMetricsAggregation"}
{"level":"info","ts":1591281005.9492555,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1591281008.7723699,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1591281008.7909398,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1591281022.878182,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"kubevirt-ssp-operator-metrics","Service.Namespace":"openshift-cnv"}
{"level":"info","ts":1591281022.879933,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1591281022.880182,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1591281022.9803753,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirtcommontemplatesbundle-controller"}
{"level":"info","ts":1591281022.9804351,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirtnodelabellerbundle-controller"}
{"level":"info","ts":1591281022.9804604,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirttemplatevalidator-controller"}
{"level":"info","ts":1591281022.9804688,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"kubevirtmetricsaggregation-controller"}
{"level":"info","ts":1591281023.0806372,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirtcommontemplatesbundle-controller","worker count":1}
{"level":"info","ts":1591281023.080771,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirtnodelabellerbundle-controller","worker count":1}
{"level":"info","ts":1591281023.0811253,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirttemplatevalidator-controller","worker count":1}
{"level":"info","ts":1591281023.081312,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"kubevirtmetricsaggregation-controller","worker count":1}
{"level":"info","ts":1591281025.108222,"logger":"logging_event_handler","msg":"[playbook task]","name":"common-templates-kubevirt-hyperconverged","namespace":"openshift","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtCommonTemplatesBundle","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"Gathering Facts"}
{"level":"info","ts":1591281025.2750564,"logger":"logging_event_handler","msg":"[playbook task]","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"2775422040480279449","EventData.Name":"Gathering Facts"}
{"level":"info","ts":1591281025.2882504,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"894385949183117216","EventData.Name":"Gathering Facts"}
{"level":"error","ts":1591281027.020319,"logger":"logging_event_handler","msg":"","name":"metrics-aggregation-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtMetricsAggregation","event_type":"runner_on_failed","job":"4751997750760398084","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281027.2420487,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: metricsaggregation.yaml **********************************************\n\u001b[0;34m1 plays in /opt/ansible/metricsaggregation.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtMetricsAggregation by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : ok=0    changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"4751997750760398084","name":"metrics-aggregation-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
{"level":"error","ts":1591281030.548158,"logger":"logging_event_handler","msg":"","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"2775422040480279449","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281030.7633631,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=1   \u001b[0m changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"2775422040480279449","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
{"level":"error","ts":1591281030.874135,"logger":"logging_event_handler","msg":"","name":"common-templates-kubevirt-hyperconverged","namespace":"openshift","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtCommonTemplatesBundle","event_type":"runner_on_failed","job":"7504504064263669287","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281030.887357,"logger":"logging_event_handler","msg":"","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"runner_on_failed","job":"894385949183117216","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281031.1060274,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: commontemplatesbundle.yaml *******************************************\n\u001b[0;34m1 plays in /opt/ansible/commontemplatesbundle.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/commontemplatesbundle.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtCommonTemplatesBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=1   \u001b[0m changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"7504504064263669287","name":"common-templates-kubevirt-hyperconverged","namespace":"openshift","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
{"level":"error","ts":1591281031.1152005,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: templatevalidator.yaml ***********************************************\n\u001b[0;34m1 plays in /opt/ansible/templatevalidator.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[1;30mtask path: /opt/ansible/templatevalidator.yaml:1\u001b[0m\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/templatevalidator.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtTemplateValidator by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=1   \u001b[0m changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"894385949183117216","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
{"level":"info","ts":1591281032.64987,"logger":"logging_event_handler","msg":"[playbook task]","name":"kubevirt-node-labeller-bundle","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"1976235410884491574","EventData.Name":"Gathering Facts"}
{"level":"error","ts":1591281037.57158,"logger":"logging_event_handler","msg":"","name":"kubevirt-node-labeller-bundle","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"1976235410884491574","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281037.7539446,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=1   \u001b[0m changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"1976235410884491574","name":"kubevirt-node-labeller-bundle","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
{"level":"info","ts":1591281672.1888278,"logger":"logging_event_handler","msg":"[playbook task]","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"3328451335138149956","EventData.Name":"Gathering Facts"}
{"level":"error","ts":1591281677.012553,"logger":"logging_event_handler","msg":"","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"3328451335138149956","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281677.191724,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=1   \u001b[0m changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"3328451335138149956","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}
{"level":"info","ts":1591281744.8202064,"logger":"logging_event_handler","msg":"[playbook task]","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"playbook_on_task_start","job":"5263531936693774911","EventData.Name":"Gathering Facts"}
{"level":"error","ts":1591281749.7325191,"logger":"logging_event_handler","msg":"","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"ssp.kubevirt.io/v1, Kind=KubevirtNodeLabellerBundle","event_type":"runner_on_failed","job":"5263531936693774911","EventData.Task":"Extract the CR info","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}
{"level":"error","ts":1591281749.9451694,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.9\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\r\nPLAYBOOK: kubevirtnodelabeller.yaml ********************************************\n\u001b[0;34m1 plays in /opt/ansible/kubevirtnodelabeller.yaml\u001b[0m\n\r\nPLAY [localhost] ***************************************************************\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\r\n\r\nTASK [Gathering Facts] *********************************************************\r\n\u001b[1;30mtask path: /opt/ansible/kubevirtnodelabeller.yaml:1\u001b[0m\n\u001b[0;32mok: [localhost]\u001b[0m\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [KubevirtCircuitBreaker : Extract the CR info] ****************************\r\n\u001b[1;30mtask path: /opt/ansible/roles/KubevirtCircuitBreaker/tasks/main.yml:3\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Failed to find exact match for v1.KubevirtNodeLabellerBundle by [kind, name, singularName, shortNames]\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=1   \u001b[0m changed=0    unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    rescued=0    ignored=0   \r\n\n","job":"5263531936693774911","name":"node-labeller-kubevirt-hyperconverged","namespace":"openshift-cnv","error":"exit status 2","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:199"}

Comment 3 Dan Kenigsberg 2020-06-10 08:08:58 UTC
Requesting blocker+ flag, as upgrade to 2.4 must succeed.

Comment 4 Omer Yahud 2020-06-15 10:01:24 UTC
A PR is open and can be followed here: https://github.com/MarSik/kubevirt-ssp-operator/pull/186

Comment 6 Omer Yahud 2020-06-28 12:30:15 UTC
SSP PR is merged, HCO PR can be found here: https://github.com/kubevirt/hyperconverged-cluster-operator/pull/634

Comment 7 Omer Yahud 2020-06-28 12:31:24 UTC
After the HCO team finishes their testing, both HCO and SSP operator will create d/s builds, as they are both required for the complete solution.

Comment 8 Ruth Netser 2020-07-08 12:43:15 UTC
Verified:  OCP 4.5.0-rc., upgrade from 2.3.0 - > 2.4.0 (SSP v2.4.0-66) ends successfully.
Both virt-template-validator and kubevirt-node-labeller pods are upgraded.

Comment 11 errata-xmlrpc 2020-07-28 19:10:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3194


Note You need to log in before you can comment on or make changes to this bug.