Bug 2111075

Summary: 4.9 to 4.10 upgrade path blocked due to bug in NMState pods
Product: Container Native Virtualization (CNV) Reporter: Krzysztof Majcher <kmajcher>
Component: InstallationAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED ERRATA QA Contact: Debarati Basu-Nag <dbasunag>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.10.4CC: aamirian, cnv-qe-bugs, dbasunag, kmajcher, kmcdonald, ngavrilo, ocohen, phoracek, stirabos
Target Milestone: ---   
Target Release: 4.10.4   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: CNV v4.10.4-4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2110783 Environment:
Last Closed: 2022-08-17 13:51:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2110783, 2110905    
Bug Blocks:    

Description Krzysztof Majcher 2022-07-26 12:38:25 UTC
+++ This bug was initially created as a clone of Bug #2110783 +++

Description of problem:
As described in the following bugzilla, NMState pods not compatible with Calico CNI provider preventing deploy of Openshift Virtualization. 
https://bugzilla.redhat.com/show_bug.cgi?id=2078247

This was fixed in 4.10 but the fix needs to be backported to 4.9 to allow upgrades from 4.8 to 4.10

Version-Release number of selected component (if applicable):
4.9.5

How reproducible:
Every time

Steps to Reproduce:
1. Deploy or upgrade OpenShift Virtualization 4.9 to any cluster with Calico CNI

Actual results:
Deploy and upgrade fail due to nmstate pod

Expected results:
Deploy and Upgrades are successfull

Additional info:

Previously upgrades paths were created for client to upgrade from 4.8.4 -> 4.9.5 -> 4.10.3


Once this fix is done the same upgrade path should be open for the new releases. 4.8.4 -> 4.9.6 -> 4.10.4

Comment 2 Oren Cohen 2022-07-31 09:04:13 UTC
4.9.6 --> 4.10.4 direct upgrade path is now in place, starting from bundle version v4.10.4-4

Comment 3 Debarati Basu-Nag 2022-08-09 19:41:43 UTC
Validated direct upgrade path exists between 4.9.6->4.10.4
============
[cnv-qe-jenkins@c01-dbn-496-pkzxb-executor ~]$ kubectl get csv -n openshift-cnv
I0809 19:40:34.061600  102380 request.go:665] Waited for 1.156881918s due to client-side throttling, not priority and fairness, request: GET:https://api.c01-dbn-496.cnv-qe.rhcloud.com:6443/apis/ingress.operator.openshift.io/v1?timeout=32s
NAME                                       DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v4.10.4   OpenShift Virtualization   4.10.4    kubevirt-hyperconverged-operator.v4.9.6   Succeeded
[cnv-qe-jenkins@c01-dbn-496-pkzxb-executor ~]$ kubectl get nodes
NAME                               STATUS   ROLES    AGE     VERSION
c01-dbn-496-pkzxb-master-0         Ready    master   7d19h   v1.23.5+012e945
c01-dbn-496-pkzxb-master-1         Ready    master   7d19h   v1.23.5+012e945
c01-dbn-496-pkzxb-master-2         Ready    master   7d19h   v1.23.5+012e945
c01-dbn-496-pkzxb-worker-0-7zll9   Ready    worker   7d19h   v1.23.5+012e945
c01-dbn-496-pkzxb-worker-0-9d8hn   Ready    worker   7d19h   v1.23.5+012e945
c01-dbn-496-pkzxb-worker-0-cctf2   Ready    worker   7d19h   v1.23.5+012e945
[cnv-qe-jenkins@c01-dbn-496-pkzxb-executor ~]$ kubectl get pod -n openshift-cnv
NAME                                                   READY   STATUS    RESTARTS   AGE
bridge-marker-6wwzm                                    1/1     Running   0          52m
bridge-marker-76mvc                                    1/1     Running   0          52m
bridge-marker-7nzgv                                    1/1     Running   0          52m
bridge-marker-8vvhz                                    1/1     Running   0          52m
bridge-marker-cjngm                                    1/1     Running   0          52m
bridge-marker-j6tlq                                    1/1     Running   0          52m
cdi-apiserver-548bdb8f86-2n5nq                         1/1     Running   0          52m
cdi-deployment-785545f8c5-xxf4z                        1/1     Running   0          52m
cdi-operator-786686cf98-cxwbq                          1/1     Running   0          54m
cdi-uploadproxy-b8d957576-fkvbh                        1/1     Running   0          52m
cluster-network-addons-operator-d89b69886-6r9bf        1/1     Running   0          54m
hco-operator-5f6fb7949-55kgf                           1/1     Running   0          55m
hco-webhook-786478bfc6-xgfhr                           1/1     Running   0          52m
hostpath-provisioner-csi-5b4lx                         4/4     Running   0          54m
hostpath-provisioner-csi-6ztnc                         4/4     Running   0          54m
hostpath-provisioner-csi-nhlcd                         4/4     Running   0          54m
hostpath-provisioner-fkx7m                             1/1     Running   0          54m
hostpath-provisioner-gl7xt                             1/1     Running   0          54m
hostpath-provisioner-operator-6795b45c47-2fjd6         1/1     Running   0          54m
hostpath-provisioner-qdtw2                             1/1     Running   0          54m
hyperconverged-cluster-cli-download-7576c54cdf-tscdf   1/1     Running   0          55m
kube-cni-linux-bridge-plugin-77mtk                     1/1     Running   0          49m
kube-cni-linux-bridge-plugin-9zgds                     1/1     Running   0          49m
kube-cni-linux-bridge-plugin-h85dv                     1/1     Running   0          50m
kube-cni-linux-bridge-plugin-lhb8f                     1/1     Running   0          51m
kube-cni-linux-bridge-plugin-lsx2t                     1/1     Running   0          50m
kube-cni-linux-bridge-plugin-vsfjp                     1/1     Running   0          52m
kubemacpool-cert-manager-6f7fb46b7c-skwzc              1/1     Running   0          52m
kubemacpool-mac-controller-manager-6f645b6c6b-27qjw    1/1     Running   0          52m
nmstate-cert-manager-6f6758db8-mf2bm                   1/1     Running   0          52m
nmstate-handler-df72t                                  1/1     Running   0          51m
nmstate-handler-hrm7k                                  1/1     Running   0          52m
nmstate-handler-mmdw2                                  1/1     Running   0          52m
nmstate-handler-q2khx                                  1/1     Running   0          51m
nmstate-handler-tsft9                                  1/1     Running   0          51m
nmstate-handler-twfgp                                  1/1     Running   0          52m
nmstate-webhook-66f7fd7ddb-t4l6n                       1/1     Running   0          52m
nmstate-webhook-66f7fd7ddb-v9jmh                       1/1     Running   0          52m
node-maintenance-operator-748489d7f8-m4xns             1/1     Running   0          52m
ssp-operator-55bbb95d5-8b52w                           1/1     Running   0          52m
virt-api-7c5877664c-m9xnt                              1/1     Running   0          48m
virt-api-7c5877664c-xfx9w                              1/1     Running   0          49m
virt-controller-d767968dc-hdbc9                        1/1     Running   0          50m
virt-controller-d767968dc-hpcqx                        1/1     Running   0          49m
virt-handler-bplzr                                     1/1     Running   0          51m
virt-handler-fftq6                                     1/1     Running   0          52m
virt-handler-jwrwp                                     1/1     Running   0          50m
virt-operator-5b84c568b6-7hs8m                         1/1     Running   0          55m
virt-operator-5b84c568b6-bq9vm                         1/1     Running   0          53m
virt-template-validator-77dcbf996c-54kwd               1/1     Running   0          53m
virt-template-validator-77dcbf996c-rvc22               1/1     Running   0          54m
[cnv-qe-jenkins@c01-dbn-496-pkzxb-executor ~]$

Comment 9 errata-xmlrpc 2022-08-17 13:51:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.10.4 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6106