Bug 1947566
| Summary: | Unable to upgrade cluster version | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Audrey.Holloman |
| Component: | Cluster Version Operator | Assignee: | Over the Air Updates <aos-team-ota> |
| Status: | CLOSED DUPLICATE | QA Contact: | Johnny Liu <jialiu> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.3.z | CC: | aos-bugs, jokerman, vrutkovs, wking |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Mac OS | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-04-08 20:26:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Please attach must-gather archive - see https://docs.openshift.com/container-platform/4.4/support/gathering-cluster-data.html > # oc get clusterversion > NAME VERSION AVAILABLE PROGRESSING SINCE STATUS > version 4.3.13 True True 6d3h Working towards 4.5.36: 84% complete This sounds a lot like you got bit by bug 1802553. I'm closing this as a dup of that one, but that doesn't help your current cluster. If you point us at the must-gather Vadim calls for in comment 1, we may be able to help unstick you. If it's a supported cluster (although 4.3 has been end-of-life for a while now [1]), you can also open a support case and get help recovering the cluster there too. You can confirm that this is a dup of bug 1802553 by looking for Partial vs. Completed updates in ClusterVersion's status.history. If you did not complete a 4.4 release before retargeting to 4.5, that would be bug 1802553. If you do see Completed 4.4 releases, then please re-open this bug. [1]: https://access.redhat.com/support/policy/updates/openshift#dates *** This bug has been marked as a duplicate of bug 1802553 *** I am confirming this issue can be closed. It appears to be a duplicate of bug 1802553. Thanks |
I have OpenShift v4.3.13 installed with 3 masters and 5 workers. I want to upgrade to the latest version. First, I tried upgrading to latest version on current 4.3 channel # oc adm upgrade --to-latest Then, I tried switching to the next channel (4.4) and upgrade to latest on new channel # oc patch clusterversion/version -p '{"spec":{"channel":"stable-4.4"}}' --type=merge # oc adm upgrade # oc adm upgrade --to-latest I repeated for 4.4 to 4.5 # oc patch clusterversion/version -p '{"spec":{"channel":"stable-4.5"}}' --# type=merge # oc adm upgrade # oc adm upgrade --to-latest Then I noticed it wasn't upgrading correctly. The status shown below is "Unable to apply 4.5.36: the cluster operator machine-config has not yet successfully rolled out". Now my cluster keeps going through a continuous loop of trying to perform an upgrade from 4.3.13 directly to 4.5.36. # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.13 True True 6d3h Unable to apply 4.5.36: the cluster operator machine-config has not yet successfully rolled out After a few minutes, the upgrade will try again and stay in a status show below of "Working towards 4.5.36: 84% complete" for a while until it returns back to the status shown above of "Unable to apply 4.5.36: the cluster operator machine-config has not yet successfully rolled out". # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.13 True True 6d3h Working towards 4.5.36: 84% complete Additional cluster version information: # oc version Client Version: 4.3.13 Server Version: 4.5.36 Kubernetes Version: v1.18.3+cdb0358 # oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.36 True False False 6d21h cloud-credential 4.5.36 True False False 6d21h cluster-autoscaler 4.5.36 True False False 6d21h config-operator 4.5.36 True False False 3d5h console 4.5.36 True False False 26h csi-snapshot-controller 4.5.36 True False False 25h dns 4.5.36 True False False 6d21h etcd 4.5.36 True False False 6d2h image-registry 4.5.36 True False False 26h ingress 4.5.36 True False False 6d21h insights 4.5.36 True False False 6d21h kube-apiserver 4.5.36 True False False 6d21h kube-controller-manager 4.5.36 True False False 6d2h kube-scheduler 4.5.36 True False False 6d2h kube-storage-version-migrator 4.5.36 True False False 26h machine-api 4.5.36 True False False 6d21h machine-approver 4.5.36 True False False 26h machine-config 4.3.13 False True True 6d2h marketplace 4.5.36 True False False 26h monitoring 4.5.36 True False False 26h network 4.5.36 True False False 6d21h node-tuning 4.5.36 True False False 26h openshift-apiserver 4.5.36 True False False 26h openshift-controller-manager 4.5.36 True False False 6d21h openshift-samples 4.5.36 True False False 26h operator-lifecycle-manager 4.5.36 True False False 6d21h operator-lifecycle-manager-catalog 4.5.36 True False False 6d21h operator-lifecycle-manager-packageserver 4.5.36 True False False 26h service-ca 4.5.36 True False False 6d21h service-catalog-apiserver 4.3.35 True False False 6d21h service-catalog-controller-manager 4.3.35 True False False 6d21h storage 4.5.36 True False False 26h # oc get nodes NAME STATUS ROLES AGE VERSION master0.aud.os.fyre.ibm.com Ready master 6d21h v1.16.2+7279a4a master1.aud.os.fyre.ibm.com Ready master 6d21h v1.16.2+7279a4a master2.aud.os.fyre.ibm.com Ready master 6d21h v1.16.2 worker0.aud.os.fyre.ibm.com Ready worker 6d21h v1.18.3+cdb0358 worker1.aud.os.fyre.ibm.com Ready worker 6d21h v1.18.3+cdb0358 worker2.aud.os.fyre.ibm.com Ready worker 6d21h v1.18.3+cdb0358 worker3.aud.os.fyre.ibm.com Ready worker 6d21h v1.18.3+cdb0358 worker4.aud.os.fyre.ibm.com Ready worker 6d21h v1.18.3+cdb0358 # oc describe clusteroperator machine-config Name: machine-config Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2021-04-01T19:38:47Z Generation: 1 Resource Version: 3287433 Self Link: /apis/config.openshift.io/v1/clusteroperators/machine-config UID: 80f19b4c-85f6-47a2-aa57-a3f3c07ddb8c Spec: Status: Conditions: Last Transition Time: 2021-04-02T14:40:47Z Message: Cluster not available for 4.5.36 Status: False Type: Available Last Transition Time: 2021-04-02T14:18:11Z Message: Working towards 4.5.36 Status: True Type: Progressing Last Transition Time: 2021-04-02T14:40:47Z Message: Unable to apply 4.5.36: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9a8e9efac6f0fca77a9c376e4007242a expected f0034240b4ad1a5537bca817454717e418a85370 has f6d1fe753cbcecb3aa1c2d3d3edd4a5d04ffca54, retrying Reason: RequiredPoolsFailed Status: True Type: Degraded Last Transition Time: 2021-04-01T19:40:14Z Reason: AsExpected Status: True Type: Upgradeable Extension: Last Sync Error: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9a8e9efac6f0fca77a9c376e4007242a expected f0034240b4ad1a5537bca817454717e418a85370 has f6d1fe753cbcecb3aa1c2d3d3edd4a5d04ffca54, retrying Worker: all 5 nodes are at latest configuration rendered-worker-3e77c6bc0258369777986cf5d23e1ab9 Related Objects: Group: Name: openshift-machine-config-operator Resource: namespaces Group: machineconfiguration.openshift.io Name: master Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: worker Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: machine-config-controller Resource: controllerconfigs Versions: Name: operator Version: 4.3.13 Events: <none> # for node in $(oc get nodes -o name | awk -F'/' '{ print $2 }');do echo "-------------------- $node ------------------"; oc describe node $node | grep machineconfiguration.openshift.io/state; done -------------------- master0.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Degraded -------------------- master1.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Degraded -------------------- master2.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Degraded -------------------- worker0.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Done -------------------- worker1.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Done -------------------- worker2.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Done -------------------- worker3.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Done -------------------- worker4.aud.os.fyre.ibm.com ------------------ machineconfiguration.openshift.io/state: Done # oc get machineconfigpool NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-9a8e9efac6f0fca77a9c376e4007242a False True True 3 0 0 3 6d22h worker rendered-worker-3e77c6bc0258369777986cf5d23e1ab9 True False False 5 5 5 0 6d22h What should I do to fix this?