Bug 1856447

Summary: CNV upgrade - HCO fails to identify wrong observedVersion in CR, HCO is reported as READY
Product: Container Native Virtualization (CNV) Reporter: Ruth Netser <rnetser>
Component: InstallationAssignee: Nahshon Unna-Tsameret <nunnatsa>
Status: CLOSED ERRATA QA Contact: Guy Inger <ginger>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 2.4.0CC: cnv-qe-bugs, ipinto, ncredi, nunnatsa, ocohen, stirabos
Target Milestone: ---Keywords: UpgradeBlocker
Target Release: 2.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: hco-bundle-registry-container-v2.3.0-478 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-28 19:10:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ruth Netser 2020-07-13 16:36:51 UTC 
Description of problem:
Upgrade CNV 2.3.0 - > 2.4.0
Due to bug 1856438, CNAO fails to update.
HCO is Ready though hyperconverged Progressing condition is 

    - lastHeartbeatTime: "2020-07-13T14:37:18Z"
      lastTransitionTime: "2020-07-13T13:40:02Z"
      message: HCO is now upgrading to version v2.4.0
      reason: HCOUpgrading
      status: "True"
      type: Progressing


Version-Release number of selected component (if applicable):
CNV 2.3.0 - > 2.4.0

How reproducible:
100%

Steps to Reproduce:
1. OCP 4.5.0
2. Upgrade CNV 2.3.0 - > 2.4.0 (with invalid observedVersion in a CR)


Actual results:
HCO is Ready though hyperconverged Progressing condition is 

    - lastHeartbeatTime: "2020-07-13T14:37:18Z"
      lastTransitionTime: "2020-07-13T13:40:02Z"
      message: HCO is now upgrading to version v2.4.0
      reason: HCOUpgrading
      status: "True"
      type: Progressing


Expected results:
HCO should be reported as Ready only when all the operators are ready (by conditions) AND the versions match

Additional info:
$ oc get pod -n openshift-cnv|grep hco
hco-operator-6fcfdddb79-dfxgw                     1/1     Running   0          173m


$ oc get csv -A
NAMESPACE                              NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
openshift-cnv                          kubevirt-hyperconverged-operator.v2.4.0   OpenShift virtualization   2.4.0     kubevirt-hyperconverged-operator.v2.3.0   Succeeded

$ oc get hyperconverged -A -o yaml 
apiVersion: v1
items:
- apiVersion: hco.kubevirt.io/v1alpha1
  kind: HyperConverged
  metadata:
    creationTimestamp: "2020-07-13T08:38:32Z"
    finalizers:
    - hyperconvergeds.hco.kubevirt.io
    generation: 1
    labels:
      app: kubevirt-hyperconverged
    name: kubevirt-hyperconverged
    namespace: openshift-cnv
    resourceVersion: "304174"
    selfLink: /apis/hco.kubevirt.io/v1alpha1/namespaces/openshift-cnv/hyperconvergeds/kubevirt-hyperconverged
    uid: f7ab54b9-f794-4bb6-baec-b549279bc09d
  spec:
    BareMetalPlatform: true
  status:
    conditions:
    - lastHeartbeatTime: "2020-07-13T16:32:27Z"
      lastTransitionTime: "2020-07-13T08:38:32Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "True"
      type: ReconcileComplete
    - lastHeartbeatTime: "2020-07-13T16:32:27Z"
      lastTransitionTime: "2020-07-13T13:49:29Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "True"
      type: Available
    - lastHeartbeatTime: "2020-07-13T16:32:27Z"
      lastTransitionTime: "2020-07-13T13:40:02Z"
      message: HCO is now upgrading to version v2.4.0
      reason: HCOUpgrading
      status: "True"
      type: Progressing
    - lastHeartbeatTime: "2020-07-13T16:32:27Z"
      lastTransitionTime: "2020-07-13T13:45:39Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "False"
      type: Degraded
    - lastHeartbeatTime: "2020-07-13T16:32:27Z"
      lastTransitionTime: "2020-07-13T13:49:29Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "True"
      type: Upgradeable
    relatedObjects:
    - apiVersion: v1
      kind: ConfigMap
      name: kubevirt-config
      namespace: openshift-cnv
      resourceVersion: "216619"
      uid: b95861d3-6727-414e-b0df-f448257736ec
    - apiVersion: v1
      kind: ConfigMap
      name: kubevirt-storage-class-defaults
      namespace: openshift-cnv
      resourceVersion: "34410"
      uid: 802f9bfa-f88b-4bf8-bb65-a1f5d4fe3820
    - apiVersion: kubevirt.io/v1alpha3
      kind: KubeVirt
      name: kubevirt-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "221473"
      uid: e02dbebc-d366-4b68-8698-7e68aaf05588
    - apiVersion: cdi.kubevirt.io/v1alpha1
      kind: CDI
      name: cdi-kubevirt-hyperconverged
      resourceVersion: "218905"
      uid: f5a801f2-4494-46d2-ade4-b5c73204c23a
    - apiVersion: networkaddonsoperator.network.kubevirt.io/v1alpha1
      kind: NetworkAddonsConfig
      name: cluster
      resourceVersion: "304162"
      uid: 07544f38-0e5d-4502-bd89-d926247d26bd
    - apiVersion: kubevirt.io/v1
      kind: KubevirtCommonTemplatesBundle
      name: common-templates-kubevirt-hyperconverged
      namespace: openshift
      resourceVersion: "215157"
      uid: 4b49439d-3dda-4f01-a3a1-7c6dbdf4796b
    - apiVersion: kubevirt.io/v1
      kind: KubevirtNodeLabellerBundle
      name: node-labeller-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "113249"
      uid: 4be334d2-9e6b-437f-b7a5-b976ba4b6d81
    - apiVersion: kubevirt.io/v1
      kind: KubevirtTemplateValidator
      name: template-validator-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "108150"
      uid: 12e8dc10-a3c6-43a1-990e-da2ef0aee9fc
    - apiVersion: kubevirt.io/v1
      kind: KubevirtMetricsAggregation
      name: metrics-aggregation-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "102909"
      uid: 1df684e8-5322-4758-b92a-6e2e603ca114
    - apiVersion: v1
      kind: ConfigMap
      name: v2v-vmware
      namespace: openshift-cnv
      resourceVersion: "34422"
      uid: 74f8c6b2-2a9b-47e7-9505-4317c6c77992
    - apiVersion: scheduling.k8s.io/v1
      kind: PriorityClass
      name: kubevirt-cluster-critical
      resourceVersion: "216592"
      uid: a14349a0-7181-4f6e-898d-1c0a3bb5268e
    - apiVersion: ssp.kubevirt.io/v1
      kind: KubevirtCommonTemplatesBundle
      name: common-templates-kubevirt-hyperconverged
      namespace: openshift
      resourceVersion: "301639"
      uid: 0f79b278-cd10-4e7d-a10e-784749eaf423
    - apiVersion: ssp.kubevirt.io/v1
      kind: KubevirtNodeLabellerBundle
      name: node-labeller-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "219795"
      uid: 07df38af-38a5-4e3c-aa81-b0f194df25b3
    - apiVersion: ssp.kubevirt.io/v1
      kind: KubevirtTemplateValidator
      name: template-validator-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "219174"
      uid: ab393cb3-0d93-4b0b-9e38-d9642e59456b
    - apiVersion: ssp.kubevirt.io/v1
      kind: KubevirtMetricsAggregation
      name: metrics-aggregation-kubevirt-hyperconverged
      namespace: openshift-cnv
      resourceVersion: "217121"
      uid: 7f667f94-384b-4211-85af-a41c6ba07e6d
    - apiVersion: v2v.kubevirt.io/v1alpha1
      kind: VMImportConfig
      name: vmimport-kubevirt-hyperconverged
      resourceVersion: "216816"
      uid: 7c2f783d-41da-476e-9a14-265a7ef90a45
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Comment 3 Ruth Netser 2020-07-16 14:50:20 UTC 
Verified that with version failure, HCO is not ready and upgrade does not end (re-open of bug 1856438).

Before closing this bug, once bug 1856438 is verified, need to verify that upgrade ends succesfully.
Comment 4 Ruth Netser 2020-07-20 12:38:43 UTC 
Verified on OCP 4.5.0-rc7, CNV 2.3.0 -> 2.4.0 upgrade
cluster-network-addons-operator-container-v2.4.0-36
hyperconverged-cluster-operator-container-v2.4.0-68

CNV upgrade ends successfully.
HCO is running

$ oc get pod -n openshift-cnv|grep hco
hco-operator-78bb789997-x5h84                         1/1     Running   0          32m


$ oc get csv -n openshift-cnv 
NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v2.4.0   OpenShift Virtualization   2.4.0     kubevirt-hyperconverged-operator.v2.3.0   Succeeded



$ oc get hyperconvergeds.hco.kubevirt.io -oyaml -n openshift-cnv
....
....
  status:
    conditions:
    - lastHeartbeatTime: "2020-07-20T12:34:27Z"
      lastTransitionTime: "2020-07-20T09:33:13Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "True"
      type: ReconcileComplete
    - lastHeartbeatTime: "2020-07-20T12:34:27Z"
      lastTransitionTime: "2020-07-20T12:21:55Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "True"
      type: Available
    - lastHeartbeatTime: "2020-07-20T12:34:27Z"
      lastTransitionTime: "2020-07-20T12:21:54Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "False"
      type: Progressing
    - lastHeartbeatTime: "2020-07-20T12:34:27Z"
      lastTransitionTime: "2020-07-20T12:20:10Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "False"
      type: Degraded
    - lastHeartbeatTime: "2020-07-20T12:34:27Z"
      lastTransitionTime: "2020-07-20T12:21:54Z"
      message: Reconcile completed successfully
      reason: ReconcileCompleted
      status: "True"
      type: Upgradeable
Comment 7 errata-xmlrpc 2020-07-28 19:10:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3194