Bug 2019705

Summary: Can't upgrade to 2.6.7 since migration bug, we need to skip CNV 2.6.7
Product: Container Native Virtualization (CNV) Reporter: Israel Pinto <ipinto>
Component: InstallationAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED CURRENTRELEASE QA Contact: Israel Pinto <ipinto>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 2.6.8CC: cnv-qe-bugs, danken, dbasunag, kbidarka, pelauter, rnetser, stirabos
Target Milestone: ---   
Target Release: 2.6.8   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: hco-bundle-registry-container-v2.6.8-23 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-01 17:21:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Israel Pinto 2021-11-03 08:11:19 UTC 
Description of problem:
Since we have bug in migration https://bugzilla.redhat.com/show_bug.cgi?id=2013494
We can't upgrade old version: 2.5.8, 2.6.4,2.6.5,2.6.6 
To CNV 2.6.7 We will need to skip this version and direct to 2.6.8



Note that we will need to also make sure that we will upgrade from 2.6.8 to 4.8.3 not to hit 
the migration bug https://bugzilla.redhat.com/show_bug.cgi?id=2013494.

 

Version-Release number of selected component (if applicable):
2.6.7

How reproducible:
100%

Additional info:

Verification flow of the migration bug
Start with VM in 2.5.8 (CNV 2.5.8, OCP 4.6)
Do OCP upgrade to 4.7 and start CNV upgrade with the following scenarios:

Scenario 1: 

Upgrade CNV From: 2.5.8 To: 2.6.4 

Migration VM in CNV 2.6.4 (Virt-Launcher version - 2.6.4/2.6.3-2): migration succeed 

Continue the upgrade to CNV 2.6.8(Virt-Launcher version 2.6.4/2.6.3-2 to 2.6.8-5) - PASSED

Scenario 2: 

Upgrade CNV From: 2.5.8 To: 2.6.5 

Migration VM in CNV 2.6.5(Virt-Launcher version - 2.6.5-2): migration succeed 

Continue the upgrade to CNV 2.6.8(Virt-Launcher version 2.6.5-2 to 2.6.8-5) - PASSED

Scenario 3: 

Upgrade CNV From: 2.5.8 To: 2.6.6 

Migration VM in CNV 2.6.6(Virt-Launcher version - 2.6.6-7): migration succeed  

Continue the upgrade to CNV 2.6.8(Virt-Launcher version 2.6.6-7 to 2.6.8-5) - PASSED

Scenario 4: 

Upgrade CNV From: 2.5.8 To: 2.6.7 

Migration VM in CNV 2.6.7() migration Failed - hit the bug: (https://bugzilla.redhat.com/show_bug.cgi?id=2013494)
[CNV-2.6.8] VMI is in LiveMigrate loop when Upgrading Cluster from 2.6.7/4.7.32 to OCP 4.8.13
Since this bug: 
We stopped and start the VM and continue the upgrade to CNV 2.6.8 - PASSED

Scenario 5: 

Upgrade CNV From: 2.5.8 To: 2.6.8 

Continue the upgrade to CNV 2.6.8 - PASSED

Migration VM in CNV 2.6.8 migration succeed
Comment 1 Kedar Bidarkar 2021-11-03 09:32:59 UTC 
Scenario 4: 
Upgrade CNV From: 2.5.8 To: 2.6.7 
---
Migrate VM in CNV 2.6.7 LiveMigration  - FAILED 
	Virt-Launcher version 2.5.8 to 2.6.7 
	   Source Virt-Launcher Pod 2.5.8 continues to be in Running state.
           Target Virt-launcher Pod 2.6.7 enters Completed or Error state
        VMIM Object shows Status: FAILED
We hit the bug: (https://bugzilla.redhat.com/show_bug.cgi?id=2013494)

Continued upgrade to CNV 2.6.8
Upgraded the same VMI/Virt-Launcher Pod, version 2.5.8 to 2.6.8-5  - PASSED

---
Comment 2 Simone Tiraboschi 2021-11-04 09:57:55 UTC 
I just set:
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'

it will be available in the next build of the bundle.
    
QE should explicitly test:
    - 2.5.8 => 2.6.8
    - 2.6.6 => 2.6.8
    - 2.6.7 => 2.6.8

@ipinto please ack
Comment 3 Israel Pinto 2021-11-07 10:13:30 UTC 
(In reply to Simone Tiraboschi from comment #2)
> I just set:
>     olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
> 
> it will be available in the next build of the bundle.
>     
> QE should explicitly test:
>     - 2.5.8 => 2.6.8
>     - 2.6.6 => 2.6.8
>     - 2.6.7 => 2.6.8
> 
> @ipinto please ack

Debarati:

Please verify the skip range.

Simone: 
WE should also have 2.6.4 => 2.6.8 No?
Comment 5 Simone Tiraboschi 2021-11-08 11:12:56 UTC 
(In reply to Israel Pinto from comment #3)
> Simone: 
> WE should also have 2.6.4 => 2.6.8 No?

No
Comment 10 Debarati Basu-Nag 2021-11-09 01:34:31 UTC 
@ipinto Validated olm.skipRange is set right against 2.6.3-23
===========================
[cnv-qe-jenkins@c01-besso-26-pqdzz-executor ~]$ kubectl get csv -n openshift-cnv kubevirt-hyperconverged-operator.v2.6.8 -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[{"apiVersion":"hco.kubevirt.io/v1beta1","kind":"HyperConverged","metadata":{"annotations":{"deployOVS":"false"},"name":"kubevirt-hyperconverged","namespace":"openshift-cnv"},"spec":{"BareMetalPlatform":false}},{"apiVersion":"networkaddonsoperator.network.kubevirt.io/v1","kind":"NetworkAddonsConfig","metadata":{"name":"cluster"},"spec":{"imagePullPolicy":"IfNotPresent","kubeMacPool":{"rangeEnd":"FD:FF:FF:FF:FF:FF","rangeStart":"02:00:00:00:00:00"},"linuxBridge":{},"macvtap":{},"multus":{},"nmstate":{},"ovs":{}}},{"apiVersion":"kubevirt.io/v1alpha3","kind":"KubeVirt","metadata":{"name":"kubevirt","namespace":"kubevirt"},"spec":{"imagePullPolicy":"Always"}},{"apiVersion":"ssp.kubevirt.io/v1beta1","kind":"SSP","metadata":{"name":"ssp-sample","namespace":"kubevirt"},"spec":{"commonTemplates":{"namespace":"kubevirt"},"templateValidator":{"replicas":2}}},{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"CDI","metadata":{"name":"cdi","namespace":"cdi"},"spec":{"imagePullPolicy":"IfNotPresent"}},{"apiVersion":"nodemaintenance.kubevirt.io/v1beta1","kind":"NodeMaintenance","metadata":{"name":"nodemaintenance-example"},"spec":{"nodeName":"node02","reason":"Test
      node maintenance"}},{"apiVersion":"hostpathprovisioner.kubevirt.io/v1beta1","kind":"HostPathProvisioner","metadata":{"name":"hostpath-provisioner"},"spec":{"imagePullPolicy":"IfNotPresent","pathConfig":{"path":"/var/hpvolumes","useNamingPrefix":false}}},{"apiVersion":"v2v.kubevirt.io/v1beta1","kind":"VMImportConfig","metadata":{"name":"vm-import-operator-config"},"spec":{"imagePullPolicy":"IfNotPresent"}}]'
    capabilities: Full Lifecycle
    categories: OpenShift Optional
    certified: "true"
    containerImage: registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:21a379643b694951d1c703c76c4e4fb28f40b79900d1b2956633e8ef2818c3fe
    createdAt: "2021-11-04 10:17:44"
    description: Creates and maintains an OpenShift Virtualization Deployment
    olm.operatorGroup: kubevirt-hyperconverged-group
    olm.operatorNamespace: openshift-cnv
    olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8"}]'
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
    olm.targetNamespaces: openshift-cnv

===========================
[cnv-qe-jenkins@c01-besso-26-pqdzz-executor ~]$ kubectl get pods -n openshift-cnv hco-operator-855f8b96b6-wrn4n  -o yaml | grep olm.skipRange
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
[cnv-qe-jenkins@c01-besso-26-pqdzz-executor ~]$ kubectl get pods -n openshift-cnv hco-webhook-78457c84fc-x2d9t  -o yaml | grep olm.skipRange
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
[cnv-qe-jenkins@c01-besso-26-pqdzz-executor ~]$ 
===========================

Not marking this as verified as following upgrade is yet to complete:
===========================
3) 2.5.8 to 2.6.8
Comment 11 Debarati Basu-Nag 2021-11-15 15:45:39 UTC 
On an upgraded cluster:
======================
[cnv-qe-jenkins@c01-258upgr-225h6-executor ~]$ kubectl get csv -A
I1115 15:43:04.369241   81927 request.go:665] Waited for 1.157086389s due to client-side throttling, not priority and fairness, request: GET:https://api.c01-258upgr.cnv-qe.rhcloud.com:6443/apis/helm.openshift.io/v1beta1?timeout=32s
NAMESPACE                              NAME                                        DISPLAY                       VERSION              REPLACES                                  PHASE
openshift-cnv                          kubevirt-hyperconverged-operator.v2.6.8     OpenShift Virtualization      2.6.8                kubevirt-hyperconverged-operator.v2.5.8   Succeeded
openshift-local-storage                local-storage-operator.4.7.0-202110121415   Local Storage                 4.7.0-202110121415                                             Succeeded
openshift-operator-lifecycle-manager   packageserver                               Package Server                0.17.0                                                         Succeeded
openshift-storage                      ocs-operator.v4.7.5                         OpenShift Container Storage   4.7.5                ocs-operator.v4.7.4                       Succeeded
[cnv-qe-jenkins@c01-258upgr-225h6-executor ~]$
======================
[cnv-qe-jenkins@c01-258upgr-225h6-executor ~]$ kubectl get csv -n openshift-cnv kubevirt-hyperconverged-operator.v2.6.8 -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[{"apiVersion":"hco.kubevirt.io/v1beta1","kind":"HyperConverged","metadata":{"annotations":{"deployOVS":"false"},"name":"kubevirt-hyperconverged","namespace":"openshift-cnv"},"spec":{"BareMetalPlatform":false}},{"apiVersion":"networkaddonsoperator.network.kubevirt.io/v1","kind":"NetworkAddonsConfig","metadata":{"name":"cluster"},"spec":{"imagePullPolicy":"IfNotPresent","kubeMacPool":{"rangeEnd":"FD:FF:FF:FF:FF:FF","rangeStart":"02:00:00:00:00:00"},"linuxBridge":{},"macvtap":{},"multus":{},"nmstate":{},"ovs":{}}},{"apiVersion":"kubevirt.io/v1alpha3","kind":"KubeVirt","metadata":{"name":"kubevirt","namespace":"kubevirt"},"spec":{"imagePullPolicy":"Always"}},{"apiVersion":"ssp.kubevirt.io/v1beta1","kind":"SSP","metadata":{"name":"ssp-sample","namespace":"kubevirt"},"spec":{"commonTemplates":{"namespace":"kubevirt"},"templateValidator":{"replicas":2}}},{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"CDI","metadata":{"name":"cdi","namespace":"cdi"},"spec":{"imagePullPolicy":"IfNotPresent"}},{"apiVersion":"nodemaintenance.kubevirt.io/v1beta1","kind":"NodeMaintenance","metadata":{"name":"nodemaintenance-example"},"spec":{"nodeName":"node02","reason":"Test
      node maintenance"}},{"apiVersion":"hostpathprovisioner.kubevirt.io/v1beta1","kind":"HostPathProvisioner","metadata":{"name":"hostpath-provisioner"},"spec":{"imagePullPolicy":"IfNotPresent","pathConfig":{"path":"/var/hpvolumes","useNamingPrefix":false}}},{"apiVersion":"v2v.kubevirt.io/v1beta1","kind":"VMImportConfig","metadata":{"name":"vm-import-operator-config"},"spec":{"imagePullPolicy":"IfNotPresent"}}]'
    capabilities: Full Lifecycle
    categories: OpenShift Optional
    certified: "true"
    containerImage: registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:21a379643b694951d1c703c76c4e4fb28f40b79900d1b2956633e8ef2818c3fe
    createdAt: "2021-11-04 10:17:44"
    description: Creates and maintains an OpenShift Virtualization Deployment
    olm.operatorGroup: kubevirt-hyperconverged-group
    olm.operatorNamespace: openshift-cnv
    olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8"}]'
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
    olm.targetNamespaces: openshift-cnv
====================================

[cnv-qe-jenkins@c01-258upgr-225h6-executor ~]$ kubectl get pods -n openshift-cnv hco-operator-f6498bb64-vzd57  -o yaml | grep olm.skipRange
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
[cnv-qe-jenkins@c01-258upgr-225h6-executor ~]$ kubectl get pods -n openshift-cnv hco-webhook-5c86fff6fc-zgs4g  -o yaml | grep olm.skipRange
    olm.skipRange: '>=2.5.8 <2.6.0 || >=2.6.6 <2.6.8'
[cnv-qe-jenkins@c01-258upgr-225h6-executor ~]$
Comment 13 Red Hat Bugzilla 2023-09-15 01:16:56 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days