Description of problem: After SDN to OVN migration, try to rollback to SDN, rollback step fail. Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-06-23-092832 How reproducible: Always Steps to Reproduce: 1. Execute SDN to OVN migration using: https://github.com/openshift/release/blob/master/ci-operator/step-registry/ovn/sdn-migration/ovn-sdn-migration-commands.sh 2. Migration succeed. 3. Execute rollback using: https://github.com/openshift/release/blob/master/ci-operator/step-registry/ovn/sdn-migration-rollback/ovn-sdn-migration-rollback-commands.sh [weliang@weliang SDN-OVN-Migration]$ ./ovn-sdn-migration-rollback-commands.sh + set -o errexit + set -o nounset + set -o pipefail + TARGET=OpenShiftSDN + oc patch MachineConfigPool master --type=merge --patch '{"spec":{"paused":true}}' machineconfigpool.machineconfiguration.openshift.io/master patched + oc patch MachineConfigPool worker --type=merge --patch '{"spec":{"paused":true}}' machineconfigpool.machineconfiguration.openshift.io/worker patched + oc patch Network.operator.openshift.io cluster --type=merge --patch '{"spec":{"migration":null}}' network.operator.openshift.io/cluster patched + timeout 60s bash migration field is not cleaned by CNO migration field is not cleaned by CNO migration field is not cleaned by CNO migration field is not cleaned by CNO migration field is not cleaned by CNO + oc patch Network.operator.openshift.io cluster --type=merge --patch '{"spec":{"migration":{"networkType":"OpenShiftSDN"}}}' network.operator.openshift.io/cluster patched + oc patch Network.config.openshift.io cluster --type=merge --patch '{"spec":{"networkType":"OpenShiftSDN"}}' network.config.openshift.io/cluster patched + oc wait co network --for=condition=PROGRESSING=True --timeout=30s error: timed out waiting for the condition on clusteroperators/network [weliang@weliang SDN-OVN-Migration]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.11.0-0.nightly-2022-06-23-092832 True False False 5m53s baremetal 4.11.0-0.nightly-2022-06-23-092832 True False False 93m cloud-controller-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 95m cloud-credential 4.11.0-0.nightly-2022-06-23-092832 True False False 94m cluster-autoscaler 4.11.0-0.nightly-2022-06-23-092832 True False False 93m config-operator 4.11.0-0.nightly-2022-06-23-092832 True False False 94m console 4.11.0-0.nightly-2022-06-23-092832 True False False 8m10s csi-snapshot-controller 4.11.0-0.nightly-2022-06-23-092832 True False False 94m dns 4.11.0-0.nightly-2022-06-23-092832 True False False 83m etcd 4.11.0-0.nightly-2022-06-23-092832 True False False 92m image-registry 4.11.0-0.nightly-2022-06-23-092832 True False False 87m ingress 4.11.0-0.nightly-2022-06-23-092832 True False False 7m56s insights 4.11.0-0.nightly-2022-06-23-092832 True False False 88m kube-apiserver 4.11.0-0.nightly-2022-06-23-092832 True False False 90m kube-controller-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 91m kube-scheduler 4.11.0-0.nightly-2022-06-23-092832 True False False 90m kube-storage-version-migrator 4.11.0-0.nightly-2022-06-23-092832 True False False 94m machine-api 4.11.0-0.nightly-2022-06-23-092832 True False False 90m machine-approver 4.11.0-0.nightly-2022-06-23-092832 True False False 93m machine-config 4.11.0-0.nightly-2022-06-23-092832 True False False 93m marketplace 4.11.0-0.nightly-2022-06-23-092832 True False False 93m monitoring 4.11.0-0.nightly-2022-06-23-092832 True False False 78m network 4.11.0-0.nightly-2022-06-23-092832 True False True 95m Not applying unsafe configuration change: invalid configuration: [cannot change default network type when not doing migration]. Use 'oc edit network.operator.openshift.io cluster' to undo the change. node-tuning 4.11.0-0.nightly-2022-06-23-092832 True False False 93m openshift-apiserver 4.11.0-0.nightly-2022-06-23-092832 True False False 6m1s openshift-controller-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 90m openshift-samples 4.11.0-0.nightly-2022-06-23-092832 True False False 87m operator-lifecycle-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 94m operator-lifecycle-manager-catalog 4.11.0-0.nightly-2022-06-23-092832 True False False 94m operator-lifecycle-manager-packageserver 4.11.0-0.nightly-2022-06-23-092832 True False False 83m service-ca 4.11.0-0.nightly-2022-06-23-092832 True False False 94m storage 4.11.0-0.nightly-2022-06-23-092832 True False False 93m [weliang@weliang SDN-OVN-Migration]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.11.0-0.nightly-2022-06-23-092832 True False False 7m4s baremetal 4.11.0-0.nightly-2022-06-23-092832 True False False 94m cloud-controller-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 96m cloud-credential 4.11.0-0.nightly-2022-06-23-092832 True False False 95m cluster-autoscaler 4.11.0-0.nightly-2022-06-23-092832 True False False 94m config-operator 4.11.0-0.nightly-2022-06-23-092832 True False False 95m console 4.11.0-0.nightly-2022-06-23-092832 True False False 9m21s csi-snapshot-controller 4.11.0-0.nightly-2022-06-23-092832 True False False 95m dns 4.11.0-0.nightly-2022-06-23-092832 True False False 84m etcd 4.11.0-0.nightly-2022-06-23-092832 True False False 93m image-registry 4.11.0-0.nightly-2022-06-23-092832 True False False 88m ingress 4.11.0-0.nightly-2022-06-23-092832 True False False 9m7s insights 4.11.0-0.nightly-2022-06-23-092832 True False False 89m kube-apiserver 4.11.0-0.nightly-2022-06-23-092832 True False False 91m kube-controller-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 92m kube-scheduler 4.11.0-0.nightly-2022-06-23-092832 True False False 91m kube-storage-version-migrator 4.11.0-0.nightly-2022-06-23-092832 True False False 95m machine-api 4.11.0-0.nightly-2022-06-23-092832 True False False 91m machine-approver 4.11.0-0.nightly-2022-06-23-092832 True False False 94m machine-config 4.11.0-0.nightly-2022-06-23-092832 True False False 94m marketplace 4.11.0-0.nightly-2022-06-23-092832 True False False 94m monitoring 4.11.0-0.nightly-2022-06-23-092832 True False False 79m network 4.11.0-0.nightly-2022-06-23-092832 True False True 97m Not applying unsafe configuration change: invalid configuration: [cannot change default network type when not doing migration]. Use 'oc edit network.operator.openshift.io cluster' to undo the change. node-tuning 4.11.0-0.nightly-2022-06-23-092832 True False False 94m openshift-apiserver 4.11.0-0.nightly-2022-06-23-092832 True False False 7m12s openshift-controller-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 91m openshift-samples 4.11.0-0.nightly-2022-06-23-092832 True False False 88m operator-lifecycle-manager 4.11.0-0.nightly-2022-06-23-092832 True False False 95m operator-lifecycle-manager-catalog 4.11.0-0.nightly-2022-06-23-092832 True False False 95m operator-lifecycle-manager-packageserver 4.11.0-0.nightly-2022-06-23-092832 True False False 84m service-ca 4.11.0-0.nightly-2022-06-23-092832 True False False 95m storage 4.11.0-0.nightly-2022-06-23-092832 True False False 94m [weliang@weliang SDN-OVN-Migration]$ Actual results: Not applying unsafe configuration change: invalid configuration: [cannot change default network type when not doing migration]. Use 'oc edit network.operator.openshift.io cluster' to undo the change. Expected results: Rollback should succeed. Additional info:
(In reply to Weibin Liang from comment #0) > Description of problem: > After SDN to OVN migration, try to rollback to SDN, rollback step fail. > > Version-Release number of selected component (if applicable): > 4.11.0-0.nightly-2022-06-23-092832 > > > How reproducible: > Always > > Steps to Reproduce: > 1. Execute SDN to OVN migration using: > https://github.com/openshift/release/blob/master/ci-operator/step-registry/ > ovn/sdn-migration/ovn-sdn-migration-commands.sh > 2. Migration succeed. > 3. Execute rollback using: > https://github.com/openshift/release/blob/master/ci-operator/step-registry/ > ovn/sdn-migration-rollback/ovn-sdn-migration-rollback-commands.sh > [weliang@weliang SDN-OVN-Migration]$ ./ovn-sdn-migration-rollback-commands.sh > + set -o errexit > + set -o nounset > + set -o pipefail > + TARGET=OpenShiftSDN > + oc patch MachineConfigPool master --type=merge --patch > '{"spec":{"paused":true}}' > machineconfigpool.machineconfiguration.openshift.io/master patched > + oc patch MachineConfigPool worker --type=merge --patch > '{"spec":{"paused":true}}' > machineconfigpool.machineconfiguration.openshift.io/worker patched > + oc patch Network.operator.openshift.io cluster --type=merge --patch > '{"spec":{"migration":null}}' > network.operator.openshift.io/cluster patched > + timeout 60s bash > migration field is not cleaned by CNO > migration field is not cleaned by CNO > migration field is not cleaned by CNO > migration field is not cleaned by CNO > migration field is not cleaned by CNO > + oc patch Network.operator.openshift.io cluster --type=merge --patch > '{"spec":{"migration":{"networkType":"OpenShiftSDN"}}}' > network.operator.openshift.io/cluster patched > + oc patch Network.config.openshift.io cluster --type=merge --patch > '{"spec":{"networkType":"OpenShiftSDN"}}' > network.config.openshift.io/cluster patched > + oc wait co network --for=condition=PROGRESSING=True --timeout=30s > error: timed out waiting for the condition on clusteroperators/network > [weliang@weliang SDN-OVN-Migration]$ oc get co > NAME VERSION > AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE > authentication > 4.11.0-0.nightly-2022-06-23-092832 True False False > 5m53s > baremetal > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > cloud-controller-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > cloud-credential > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > cluster-autoscaler > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > config-operator > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > console > 4.11.0-0.nightly-2022-06-23-092832 True False False > 8m10s > csi-snapshot-controller > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > dns > 4.11.0-0.nightly-2022-06-23-092832 True False False > 83m > etcd > 4.11.0-0.nightly-2022-06-23-092832 True False False > 92m > image-registry > 4.11.0-0.nightly-2022-06-23-092832 True False False > 87m > ingress > 4.11.0-0.nightly-2022-06-23-092832 True False False > 7m56s > insights > 4.11.0-0.nightly-2022-06-23-092832 True False False > 88m > kube-apiserver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 90m > kube-controller-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 91m > kube-scheduler > 4.11.0-0.nightly-2022-06-23-092832 True False False > 90m > kube-storage-version-migrator > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > machine-api > 4.11.0-0.nightly-2022-06-23-092832 True False False > 90m > machine-approver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > machine-config > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > marketplace > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > monitoring > 4.11.0-0.nightly-2022-06-23-092832 True False False > 78m > network > 4.11.0-0.nightly-2022-06-23-092832 True False True > 95m Not applying unsafe configuration change: invalid configuration: > [cannot change default network type when not doing migration]. Use 'oc edit > network.operator.openshift.io cluster' to undo the change. > node-tuning > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > openshift-apiserver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 6m1s > openshift-controller-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 90m > openshift-samples > 4.11.0-0.nightly-2022-06-23-092832 True False False > 87m > operator-lifecycle-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > operator-lifecycle-manager-catalog > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > operator-lifecycle-manager-packageserver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 83m > service-ca > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > storage > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > [weliang@weliang SDN-OVN-Migration]$ oc get co > NAME VERSION > AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE > authentication > 4.11.0-0.nightly-2022-06-23-092832 True False False > 7m4s > baremetal > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > cloud-controller-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 96m > cloud-credential > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > cluster-autoscaler > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > config-operator > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > console > 4.11.0-0.nightly-2022-06-23-092832 True False False > 9m21s > csi-snapshot-controller > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > dns > 4.11.0-0.nightly-2022-06-23-092832 True False False > 84m > etcd > 4.11.0-0.nightly-2022-06-23-092832 True False False > 93m > image-registry > 4.11.0-0.nightly-2022-06-23-092832 True False False > 88m > ingress > 4.11.0-0.nightly-2022-06-23-092832 True False False > 9m7s > insights > 4.11.0-0.nightly-2022-06-23-092832 True False False > 89m > kube-apiserver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 91m > kube-controller-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 92m > kube-scheduler > 4.11.0-0.nightly-2022-06-23-092832 True False False > 91m > kube-storage-version-migrator > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > machine-api > 4.11.0-0.nightly-2022-06-23-092832 True False False > 91m > machine-approver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > machine-config > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > marketplace > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > monitoring > 4.11.0-0.nightly-2022-06-23-092832 True False False > 79m > network > 4.11.0-0.nightly-2022-06-23-092832 True False True > 97m Not applying unsafe configuration change: invalid configuration: > [cannot change default network type when not doing migration]. Use 'oc edit > network.operator.openshift.io cluster' to undo the change. > node-tuning > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > openshift-apiserver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 7m12s > openshift-controller-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 91m > openshift-samples > 4.11.0-0.nightly-2022-06-23-092832 True False False > 88m > operator-lifecycle-manager > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > operator-lifecycle-manager-catalog > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > operator-lifecycle-manager-packageserver > 4.11.0-0.nightly-2022-06-23-092832 True False False > 84m > service-ca > 4.11.0-0.nightly-2022-06-23-092832 True False False > 95m > storage > 4.11.0-0.nightly-2022-06-23-092832 True False False > 94m > [weliang@weliang SDN-OVN-Migration]$ > > > Actual results: > Not applying unsafe configuration change: invalid configuration: [cannot > change default network type when not doing migration]. Use 'oc edit > network.operator.openshift.io cluster' to undo the change. > > Expected results: > Rollback should succeed. > > Additional info: Tested in GCP
The root cause is in the script. Before executing `oc patch Network.config.openshift.io cluster --type='merge' --patch "{\"spec\":{\"networkType\":\"${TARGET}\"}}"`, we need to check if 'status.migration.networkType' becomes 'OpenShiftSDN'.
*** Bug 2100642 has been marked as a duplicate of this bug. ***
Tested and verified in 4.11.0-rc.1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069