Bug 1909975 - CNV upgrade to 2.5.3 fails because KubevirtCommonTemplatesBundle cr fails to update to the target version
Summary: CNV upgrade to 2.5.3 fails because KubevirtCommonTemplatesBundle cr fails to ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Installation
Version: 2.5.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 2.5.6
Assignee: Simone Tiraboschi
QA Contact: Satyajit Bulage
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-22 08:27 UTC by Oren Cohen
Modified: 2024-06-13 23:47 UTC (History)
10 users (show)

Fixed In Version: hyperconverged-cluster-operator:v2.5.6-3, hco-bundle-registry-container-v2.5.6-65
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-19 14:56:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt hyperconverged-cluster-operator pull 1250 0 None open WIP: workaround for SSP stuck upgrades 2021-04-14 14:31:51 UTC
Red Hat Knowledge Base (Solution) 5900361 0 None None None 2021-03-23 18:17:35 UTC
Red Hat Product Errata RHEA-2021:2045 0 None None None 2021-05-19 14:56:44 UTC

Description Oren Cohen 2020-12-22 08:27:52 UTC
*Note*: this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1904830 that was resolved in 2.5.2 and encountered again in 2.5.3.

Description of problem:
Upgrade from 2.5.2 to 2.5.3 fails due to KubevirtCommonTemplatesBundle CR not updated with the new observedVersion.

Version-Release number of selected component (if applicable):
HCO bundle: rh-osbs/container-native-virtualization-hco-bundle-registry:v2.5.3-7
SSP image: https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/kubevirt-ssp-operator/images/v2.5.2-5

How reproducible:
100%

Steps to Reproduce:
1. deploy CNV from production
2. upgrade to 2.5.3 from OSBS
3. 

Actual results:
CSV is stuck on "Installing" state due to the mentioned issue.

Expected results:
The upgrade should be completed successfully and the CSV should be in "Succeeded" phase.

Additional info:
KubevirtCommonTemplatesBundle CR yaml:

apiVersion: ssp.kubevirt.io/v1
kind: KubevirtCommonTemplatesBundle
metadata:
  selfLink: >-
    /apis/ssp.kubevirt.io/v1/namespaces/openshift/kubevirtcommontemplatesbundles/common-templates-kubevirt-hyperconverged
  resourceVersion: '348718027'
  name: common-templates-kubevirt-hyperconverged
  uid: bf89cccb-bdc6-4393-8c18-303364ab82dd
  creationTimestamp: '2020-07-30T14:46:43Z'
  generation: 1
  managedFields:
    - apiVersion: ssp.kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:app': {}
        'f:spec': {}
      manager: hyperconverged-cluster-operator
      operation: Update
      time: '2020-07-30T14:46:43Z'
    - apiVersion: ssp.kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:observedVersion': {}
          'f:operatorVersion': {}
          'f:targetVersion': {}
      manager: OpenAPI-Generator
      operation: Update
      time: '2020-12-17T21:00:30Z'
    - apiVersion: ssp.kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:conditions': {}
      manager: ansible-operator
      operation: Update
      time: '2020-12-22T08:10:48Z'
  namespace: openshift
  labels:
    app: kubevirt-hyperconverged
spec: {}
status:
  conditions:
    - lastTransitionTime: '2020-12-17T21:00:31Z'
      message: >-
        Templates progressing (deployed templates: 148, desired deployed
        templated: 58).
      reason: progressing
      status: 'False'
      type: Progressing
    - lastTransitionTime: '2020-12-17T21:00:31Z'
      message: >-
        Common templates available (deployed templates: 148, desired deployed
        templated: 58).
      reason: available
      status: 'True'
      type: Available
    - lastTransitionTime: '2020-12-17T21:00:31Z'
      message: >-
        Templates degraded (deployed templates: 148, desired deployed templated:
        58).
      reason: degraded
      status: 'False'
      type: Degraded
    - lastTransitionTime: '2020-12-22T08:10:12Z'
      message: Running reconciliation
      reason: Running
      status: 'False'
      type: Running
    - lastTransitionTime: '2020-12-22T08:10:48Z'
      message: >-
        An unhandled exception occurred while running the lookup plugin 'k8s'.
        Error was a <class 'ansible.errors.AnsibleError'>, original message:
        Failed to find exact match for
        kubevirt.io/v1.KubevirtCommonTemplatesBundle by [kind, name,
        singularName, shortNames]
      reason: Failed
      status: 'True'
      type: Failure
  observedVersion: v2.5.2
  operatorVersion: v2.5.3
  targetVersion: v2.5.3

Comment 1 Oren Cohen 2020-12-22 09:24:19 UTC
It doesn't reproduce on a fresh cluster starting in production 2.5.2.
On the cluster in question, I resolved the issue by deleting the CR, and it was recreated with the correct observedVersion (2.5.3).

Comment 10 Ruth Netser 2021-04-13 11:10:43 UTC
Could not reporoduce the issue:

Tried to reproduce by scaling down virt-api.
The error about ""unable to retrieve the complete list of server APIs" appears in SSP log but once the api is back up, the error stops.

Comment 11 Simone Tiraboschi 2021-04-14 11:19:59 UTC
Let's try automating the workaround on HCO side.

Comment 14 Satyajit Bulage 2021-04-16 14:27:31 UTC
As per comment 12 and 13,

Verifying this BZ

Comment 19 errata-xmlrpc 2021-05-19 14:56:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.6 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2045


Note You need to log in before you can comment on or make changes to this bug.