Bug 1834925
| Summary: | [vsphere] upgrade from 4.1 -> 4.2 -> 4.3 -> 4.4 upgrade failed at waitForControllerConfigToBeCompleted | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Joseph Callen <jcallen> | |
| Component: | Machine Config Operator | Assignee: | Joseph Callen <jcallen> | |
| Machine Config Operator sub component: | platform-vsphere | QA Contact: | jima | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | high | |||
| Priority: | high | CC: | aos-bugs, ChetRHosey, jcallen, jerzhang, kgarriso, lmohanty, malonso, mkrejci, mnguyen, rkshirsa, wking, yapei | |
| Version: | 4.5 | Keywords: | Upgrades | |
| Target Milestone: | --- | |||
| Target Release: | 4.5.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1834194 | |||
| : | 1842952 (view as bug list) | Environment: | ||
| Last Closed: | 2020-07-13 17:37:59 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1842952 | |||
| Bug Blocks: | 1834194 | |||
|
Description
Joseph Callen
2020-05-12 16:53:46 UTC
Version: 4.5.0-0.nightly-2020-05-19-031245 Upgrade ocp/vsphere from v4.4.4 to 4.5.0-0.nightly-2020-05-19-031245 successfully. Machine-config-operator works well. # ./oc get co machine-config NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE machine-config 4.5.0-0.nightly-2020-05-19-031245 True False False 12h This is a high priority bug it effects upgrade of OpenShift cluster to 4.4. Hence increasing the sev and adding upgradeblocker keyword. The bug for 4.4 backport also have similar severity https://bugzilla.redhat.com/show_bug.cgi?id=1834194#c11 Is there a manual workaround for the issue which customer can do and get unblocked? Moving back to assigned based on 4.4 testing The customer cases attached to this BZ should be on: https://bugzilla.redhat.com/show_bug.cgi?id=1834194 Sorry for the inconvenience this has caused. The templates that we added to MCO in version 4.4 were the precursor to enabling vSphere IPI. The check on various variables was to ensure the difference between UPI and IPI. In our backlog we have a story (https://issues.redhat.com/browse/SPLAT-26) to implement a CI job that will upgrade a cluster on vSphere through the releases in the hope that we would catch this failure before a customer does. Workaround: 1.) oc patch infrastructure cluster --type json -p '[{"op": "add", "path": "/status/platformStatus", "value": {"type": "VSphere"}}]' 2.) oc get controllerconfigs.machineconfiguration.openshift.io machine-config-controller -o yaml > mcc.yaml 3.) oc delete controllerconfigs.machineconfiguration.openshift.io machine-config-controller 4.) confirm the above ^ is regenerated `oc get controllerconfigs.machineconfiguration.openshift.io machine-config-controller` 5.) Then perform an update verified on 4.5.0-0.nightly-2020-06-18-114733 upgrade ocp from 4.4.0-0.nightly-2020-06-18-212632 to 4.5.0-0.nightly-2020-06-18-114733, it is successful and machine-config-operator works well. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-06-18-114733 True False 20m Cluster version is 4.5.0-0.nightly-2020-06-18-114733 $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.0-0.nightly-2020-06-18-114733 True False False 98m cloud-credential 4.5.0-0.nightly-2020-06-18-114733 True False False 114m cluster-autoscaler 4.5.0-0.nightly-2020-06-18-114733 True False False 105m config-operator 4.5.0-0.nightly-2020-06-18-114733 True False False 73m console 4.5.0-0.nightly-2020-06-18-114733 True False False 33m csi-snapshot-controller 4.5.0-0.nightly-2020-06-18-114733 True False False 103m dns 4.5.0-0.nightly-2020-06-18-114733 True False False 109m etcd 4.5.0-0.nightly-2020-06-18-114733 True False False 109m image-registry 4.5.0-0.nightly-2020-06-18-114733 True False False 33m ingress 4.5.0-0.nightly-2020-06-18-114733 True False False 103m insights 4.5.0-0.nightly-2020-06-18-114733 True False False 106m kube-apiserver 4.5.0-0.nightly-2020-06-18-114733 True False False 109m kube-controller-manager 4.5.0-0.nightly-2020-06-18-114733 True False False 108m kube-scheduler 4.5.0-0.nightly-2020-06-18-114733 True False False 108m kube-storage-version-migrator 4.5.0-0.nightly-2020-06-18-114733 True False False 33m machine-api 4.5.0-0.nightly-2020-06-18-114733 True False False 106m machine-approver 4.5.0-0.nightly-2020-06-18-114733 True False False 65m machine-config 4.5.0-0.nightly-2020-06-18-114733 True False False 109m marketplace 4.5.0-0.nightly-2020-06-18-114733 True False False 32m monitoring 4.5.0-0.nightly-2020-06-18-114733 True False False 62m network 4.5.0-0.nightly-2020-06-18-114733 True False False 111m node-tuning 4.5.0-0.nightly-2020-06-18-114733 True False False 65m openshift-apiserver 4.5.0-0.nightly-2020-06-18-114733 True False False 106m openshift-controller-manager 4.5.0-0.nightly-2020-06-18-114733 True False False 106m openshift-samples 4.5.0-0.nightly-2020-06-18-114733 True False False 65m operator-lifecycle-manager 4.5.0-0.nightly-2020-06-18-114733 True False False 109m operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-06-18-114733 True False False 109m operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-06-18-114733 True False False 32m service-ca 4.5.0-0.nightly-2020-06-18-114733 True False False 110m storage 4.5.0-0.nightly-2020-06-18-114733 True False False 65m Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |