Bug 1909975

Summary: CNV upgrade to 2.5.3 fails because KubevirtCommonTemplatesBundle cr fails to update to the target version
Product: Container Native Virtualization (CNV) Reporter: Oren Cohen <ocohen>
Component: InstallationAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED ERRATA QA Contact: Satyajit Bulage <sbulage>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 2.5.3CC: bhershbe, cnv-qe-bugs, danken, dcritch, fdeutsch, kwiesmul, mmethot, oyahud, rnetser, stirabos
Target Milestone: ---Keywords: Reopened
Target Release: 2.5.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: hyperconverged-cluster-operator:v2.5.6-3, hco-bundle-registry-container-v2.5.6-65 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-19 14:56:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Oren Cohen 2020-12-22 08:27:52 UTC 
*Note*: this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1904830 that was resolved in 2.5.2 and encountered again in 2.5.3.

Description of problem:
Upgrade from 2.5.2 to 2.5.3 fails due to KubevirtCommonTemplatesBundle CR not updated with the new observedVersion.

Version-Release number of selected component (if applicable):
HCO bundle: rh-osbs/container-native-virtualization-hco-bundle-registry:v2.5.3-7
SSP image: https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/kubevirt-ssp-operator/images/v2.5.2-5

How reproducible:
100%

Steps to Reproduce:
1. deploy CNV from production
2. upgrade to 2.5.3 from OSBS
3. 

Actual results:
CSV is stuck on "Installing" state due to the mentioned issue.

Expected results:
The upgrade should be completed successfully and the CSV should be in "Succeeded" phase.

Additional info:
KubevirtCommonTemplatesBundle CR yaml:

apiVersion: ssp.kubevirt.io/v1
kind: KubevirtCommonTemplatesBundle
metadata:
  selfLink: >-
    /apis/ssp.kubevirt.io/v1/namespaces/openshift/kubevirtcommontemplatesbundles/common-templates-kubevirt-hyperconverged
  resourceVersion: '348718027'
  name: common-templates-kubevirt-hyperconverged
  uid: bf89cccb-bdc6-4393-8c18-303364ab82dd
  creationTimestamp: '2020-07-30T14:46:43Z'
  generation: 1
  managedFields:
    - apiVersion: ssp.kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:app': {}
        'f:spec': {}
      manager: hyperconverged-cluster-operator
      operation: Update
      time: '2020-07-30T14:46:43Z'
    - apiVersion: ssp.kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:observedVersion': {}
          'f:operatorVersion': {}
          'f:targetVersion': {}
      manager: OpenAPI-Generator
      operation: Update
      time: '2020-12-17T21:00:30Z'
    - apiVersion: ssp.kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:conditions': {}
      manager: ansible-operator
      operation: Update
      time: '2020-12-22T08:10:48Z'
  namespace: openshift
  labels:
    app: kubevirt-hyperconverged
spec: {}
status:
  conditions:
    - lastTransitionTime: '2020-12-17T21:00:31Z'
      message: >-
        Templates progressing (deployed templates: 148, desired deployed
        templated: 58).
      reason: progressing
      status: 'False'
      type: Progressing
    - lastTransitionTime: '2020-12-17T21:00:31Z'
      message: >-
        Common templates available (deployed templates: 148, desired deployed
        templated: 58).
      reason: available
      status: 'True'
      type: Available
    - lastTransitionTime: '2020-12-17T21:00:31Z'
      message: >-
        Templates degraded (deployed templates: 148, desired deployed templated:
        58).
      reason: degraded
      status: 'False'
      type: Degraded
    - lastTransitionTime: '2020-12-22T08:10:12Z'
      message: Running reconciliation
      reason: Running
      status: 'False'
      type: Running
    - lastTransitionTime: '2020-12-22T08:10:48Z'
      message: >-
        An unhandled exception occurred while running the lookup plugin 'k8s'.
        Error was a <class 'ansible.errors.AnsibleError'>, original message:
        Failed to find exact match for
        kubevirt.io/v1.KubevirtCommonTemplatesBundle by [kind, name,
        singularName, shortNames]
      reason: Failed
      status: 'True'
      type: Failure
  observedVersion: v2.5.2
  operatorVersion: v2.5.3
  targetVersion: v2.5.3
Comment 1 Oren Cohen 2020-12-22 09:24:19 UTC 
It doesn't reproduce on a fresh cluster starting in production 2.5.2.
On the cluster in question, I resolved the issue by deleting the CR, and it was recreated with the correct observedVersion (2.5.3).
Comment 10 Ruth Netser 2021-04-13 11:10:43 UTC 
Could not reporoduce the issue:

Tried to reproduce by scaling down virt-api.
The error about ""unable to retrieve the complete list of server APIs" appears in SSP log but once the api is back up, the error stops.
Comment 11 Simone Tiraboschi 2021-04-14 11:19:59 UTC 
Let's try automating the workaround on HCO side.
Comment 14 Satyajit Bulage 2021-04-16 14:27:31 UTC 
As per comment 12 and 13,

Verifying this BZ
Comment 19 errata-xmlrpc 2021-05-19 14:56:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.6 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2045