Bug 2080918

Summary: Upgrade CNV from 4.10.1 to 4.11 should be blocked if CNV k8s-nmstate is still installed
Product: Container Native Virtualization (CNV) Reporter: Yossi Segev <ysegev>
Component: InstallationAssignee: Nahshon Unna-Tsameret <nunnatsa>
Status: CLOSED ERRATA QA Contact: Debarati Basu-Nag <dbasunag>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.10.1CC: cnv-qe-bugs, dbasunag, myakove, nrozen, phoracek, rhrazdil, stirabos
Target Milestone: ---   
Target Release: 4.10.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: hco-bundle-registry-container-v4.10.2-5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-14 17:42:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yossi Segev 2022-05-02 11:11:48 UTC 
Description of problem:
As part of migrating k8s-nmstate from CNV to be a standalone OCP operator (https://issues.redhat.com/browse/CNV-7568), upgrading CNV 4.10.1 to CNV 4.11 should be blocked if k8s-nmstate is still installed as part of CNV (i.e. its components, e.g. nmstate-handler DaemonSet, are installed in openshift-cnv namespace).


Version-Release number of selected component (if applicable):
4.10.1


How reproducible:
Tried only once until now


Steps to Reproduce:
1. In a cluster with OCP 4.11 (after upgrading from 4.10) and CNV 4.10.1 - try upgrading CNV to 4.11 (instructions in the following comment, which will be private).


Actual results:
Upgrade succeeds, and CNV 4.11 is installed.


Expected results:
Upgrade should be blocked until CNV knmstate is removed, and standalone knmstate operator (KNO) is installed.


Additional info:
Comment 2 Debarati Basu-Nag 2022-06-03 13:57:57 UTC 
On a 4.10.2 cluster post upgrade of ocp, I see this:
====================================================
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[
  {
    "name": "operator",
    "version": "4.10.2"
  }
]
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
[
  {
    "lastTransitionTime": "2022-05-31T21:29:30Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "ReconcileComplete"
  },
  {
    "lastTransitionTime": "2022-06-03T10:27:01Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-06-03T10:27:01Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-06-03T10:27:01Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-06-02T22:00:02Z",
    "message": "NetworkAddonsConfig is not upgradeable: NMState deployment is not supported by CNAO anymore, please install Kubernetes NMState Operator",
    "observedGeneration": 2,
    "reason": "NetworkAddonsConfigNotUpgradeable",
    "status": "False",
    "type": "Upgradeable"
  }
]
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$
======================
installed kubernetes nmstate:
======================
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$ kubectl get all -n openshift-nmstate 
I0603 11:46:35.420733   96309 request.go:665] Waited for 1.143379287s due to client-side throttling, not priority and fairness, request: GET:https://api.c01-dbn-4012.cnv-qe.rhcloud.com:6443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
NAME                                        READY   STATUS    RESTARTS   AGE
pod/nmstate-cert-manager-5949c9b46d-t9tfr   1/1     Running   0          89s
pod/nmstate-handler-4pxtr                   1/1     Running   0          88s
pod/nmstate-handler-hndx7                   1/1     Running   0          88s
pod/nmstate-handler-k2ttg                   1/1     Running   0          88s
pod/nmstate-handler-k485z                   1/1     Running   0          89s
pod/nmstate-handler-m8dj5                   1/1     Running   0          88s
pod/nmstate-handler-swkf8                   1/1     Running   0          88s
pod/nmstate-operator-7646b78f5f-fgl5t       1/1     Running   0          2m5s
pod/nmstate-webhook-f965464-89frq           1/1     Running   0          89s
pod/nmstate-webhook-f965464-znkmz           1/1     Running   0          89s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/nmstate-webhook   ClusterIP   172.30.86.205   <none>        443/TCP   89s

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                          AGE
daemonset.apps/nmstate-handler   6         6         6       6            6           beta.kubernetes.io/arch=amd64,kubernetes.io/os=linux   89s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nmstate-cert-manager   1/1     1            1           89s
deployment.apps/nmstate-operator       1/1     1            1           2m5s
deployment.apps/nmstate-webhook        2/2     2            2           89s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nmstate-cert-manager-5949c9b46d   1         1         1       89s
replicaset.apps/nmstate-operator-7646b78f5f       1         1         1       2m5s
replicaset.apps/nmstate-webhook-f965464           2         2         2       89s
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$ 
==============================
Operator condition changed back to upgradable:
==============================
  status:
    conditions:
    - lastTransitionTime: "2022-06-03T11:45:07Z"
      message: ""
      observedGeneration: 7
      reason: Upgradeable
      status: "True"
      type: Upgradeable
==============================
HCO condition stabilized:
==============================
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
[
  {
    "lastTransitionTime": "2022-05-31T21:29:30Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "ReconcileComplete"
  },
  {
    "lastTransitionTime": "2022-06-03T11:45:07Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-06-03T10:27:01Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-06-03T11:45:07Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-06-03T11:45:07Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 2,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Upgradeable"
  }
]

After this cnv upgrade to 4.11 completed successfully:
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[
  {
    "name": "operator",
    "version": "4.11.0"
  }
]
[cnv-qe-jenkins@c01-dbn-4012-zxn69-executor ~]$
Comment 8 errata-xmlrpc 2022-06-14 17:42:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.2 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5026