Bug 1806483
| Summary: | cannot downgrade from 4.3.3 back to 4.2 due to machine-config operator Degraded | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | liujia <jiajliu> |
| Component: | Machine Config Operator | Assignee: | Erica von Buelow <evb> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Michael Nguyen <mnguyen> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.3.z | CC: | amurdaca |
| Target Milestone: | --- | ||
| Target Release: | 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-29 12:46:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Description of problem: Downgrade cluster from v4.3.3(has successfully upgraded from v4.2) to v4.2 nightly failed. # ./oc adm upgrade info: An upgrade is in progress. Working towards 4.2.0-0.nightly-2020-02-23-045604: 88% complete ... # ./oc adm upgrade info: An upgrade is in progress. Unable to apply 4.2.0-0.nightly-2020-02-23-045604: the cluster operator machine-config has not yet successfully rolled out # ./oc describe co machine-config Name: machine-config Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-02-24T04:06:38Z Generation: 1 Resource Version: 107233 Self Link: /apis/config.openshift.io/v1/clusteroperators/machine-config UID: 0e4ccef1-56bb-11ea-8e00-02639c396d48 Spec: Status: Conditions: Last Transition Time: 2020-02-24T07:55:56Z Message: Cluster not available for 4.2.0-0.nightly-2020-02-23-045604 Status: False Type: Available Last Transition Time: 2020-02-24T07:45:55Z Message: Working towards 4.2.0-0.nightly-2020-02-23-045604 Status: True Type: Progressing Last Transition Time: 2020-02-24T07:55:56Z Message: Unable to apply 4.2.0-0.nightly-2020-02-23-045604: timed out waiting for the condition during waitForDaemonsetRollout: Daemonset machine-config-daemon is not ready. status: (desired: 5, updated: 1, ready: 4, unavailable: 4) Reason: MachineConfigDaemonFailed Status: True Type: Degraded Last Transition Time: 2020-02-24T04:07:37Z Reason: AsExpected Status: True Type: Upgradeable Extension: Related Objects: Group: Name: openshift-machine-config-operator Resource: namespaces Group: machineconfiguration.openshift.io Name: master Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: worker Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: machine-config-controller Resource: controllerconfigs Versions: Name: operator Version: 4.3.3 Events: <none> ======================================================= One of deamon pod keep restarted. # ./oc get po -n openshift-machine-config-operator NAME READY STATUS RESTARTS AGE etcd-quorum-guard-c948f6597-dhbm9 1/1 Running 0 117m etcd-quorum-guard-c948f6597-md7ld 1/1 Running 0 117m etcd-quorum-guard-c948f6597-sjb9z 1/1 Running 0 117m machine-config-controller-7bbb6b448b-vx9xw 1/1 Running 0 126m machine-config-daemon-22w7s 2/2 Running 0 134m machine-config-daemon-8q7lm 1/2 CrashLoopBackOff 17 64m machine-config-daemon-b6gcf 2/2 Running 0 134m machine-config-daemon-dd4s9 2/2 Running 0 134m machine-config-daemon-plq45 2/2 Running 0 134m machine-config-operator-fd7d977f9-pll66 1/1 Running 0 66m machine-config-server-6rvtm 1/1 Running 0 130m machine-config-server-vfzdc 1/1 Running 0 131m machine-config-server-wt6sv 1/1 Running 0 131m # ./oc logs machine-config-daemon-8q7lm -c oauth-proxy 2020/02/24 08:12:39 provider.go:117: Defaulting client-id to system:serviceaccount:openshift-machine-config-operator:machine-config-daemon 2020/02/24 08:12:39 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token 2020/02/24 08:12:39 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates. 2020/02/24 08:12:39 main.go:138: Invalid configuration: unable to load OpenShift configuration: unable to retrieve authentication information for tokens: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:openshift-machine-config-operator:machine-config-daemon" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope Version-Release number of selected component (if applicable): 4.3.3 How reproducible: always Steps to Reproduce: 1. Install 4.2.0-0.nightly-2020-02-23-045604 2. Run upgrade from 4.2.0-0.nightly-2020-02-23-045604 to 4.3.3 succeed. 3. Run downgrade from 4.3.3 to 4.2.0-0.nightly-2020-02-23-045604 Actual results: Downgrade failed. Expected results: Downgrade succeed. Additional info: