Bug 1871068
| Summary: | [OVN]Upgrading from 4.5.6 to 4.6 latest nightly build failed | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | huirwang |
| Component: | Networking | Assignee: | Alexander Constantinescu <aconstan> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aconstan, amurdaca, anusaxen, aos-bugs, kgarriso, mfojtik, rpattath, sttts, trozet, weliang, zzhao |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-17 13:15:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Seeing the same during 4.5.7 to 4.6 nightly upgrade. Marking upgrade blocker.
# oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.6.0-0.nightly-2020-08-26-103641 False True True 18h
cloud-credential 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
cluster-autoscaler 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
config-operator 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
console 4.6.0-0.nightly-2020-08-26-103641 False False True 18h
csi-snapshot-controller 4.6.0-0.nightly-2020-08-26-103641 True False False 18h
dns 4.6.0-0.nightly-2020-08-26-103641 True False True 18h
etcd 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
image-registry 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
ingress 4.6.0-0.nightly-2020-08-26-103641 True False False 18h
insights 4.6.0-0.nightly-2020-08-26-103641 True False True 36h
kube-apiserver 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
kube-controller-manager 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
kube-scheduler 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
kube-storage-version-migrator 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
machine-api 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
machine-approver 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
machine-config 4.5.7 False True False 18h
marketplace 4.6.0-0.nightly-2020-08-26-103641 False False False 18h
monitoring 4.6.0-0.nightly-2020-08-26-103641 False True True 17h
network 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
node-tuning 4.6.0-0.nightly-2020-08-26-103641 True False False 18h
openshift-apiserver 4.6.0-0.nightly-2020-08-26-103641 False False True 138m
openshift-controller-manager 4.6.0-0.nightly-2020-08-26-103641 False True False 17h
openshift-samples 4.6.0-0.nightly-2020-08-26-103641 True False False 17h
operator-lifecycle-manager 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
operator-lifecycle-manager-catalog 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
operator-lifecycle-manager-packageserver 4.6.0-0.nightly-2020-08-26-103641 True False False 18h
service-ca 4.6.0-0.nightly-2020-08-26-103641 True False False 36h
storage 4.6.0-0.nightly-2020-08-26-103641 True False False 18h
[root@rpattath ~]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.5.7 True True 18h Unable to apply 4.6.0-0.nightly-2020-08-26-103641: the control plane is reporting an internal error
[root@rpattath ~]# oc get pods --all-namespaces | egrep -v "Running|Complete"
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-etcd etcd-quorum-guard-647745d8c-sjjmr 0/1 Pending 0 18h
[root@rpattath ~]# oc logs pod/etcd-quorum-guard-647745d8c-sjjmr -n openshift-etcd
[root@rpattath ~]# oc describe co authentication
Name: authentication
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-08-25T23:35:24Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:spec:
f:status:
.:
f:extension:
Manager: cluster-version-operator
Operation: Update
Time: 2020-08-25T23:35:24Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
f:versions:
Manager: authentication-operator
Operation: Update
Time: 2020-08-27T13:13:12Z
Resource Version: 1438927
Self Link: /apis/config.openshift.io/v1/clusteroperators/authentication
UID: 2a6a2718-474c-4429-8c5a-3a55126aa10e
Spec:
Status:
Conditions:
Last Transition Time: 2020-08-26T18:18:30Z
Message: OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Get "https://10.130.0.31:6443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
OAuthServerDeploymentDegraded: Unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io openshift-browser-client)
APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver
Reason: APIServerDeployment_UnavailablePod::OAuthRouteCheckEndpointAccessibleController_SyncError::OAuthServerDeployment_GetFailed::OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError
Status: True
Type: Degraded
Last Transition Time: 2020-08-26T18:18:29Z
Message: OAuthVersionRouteProgressing: Request to "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz" not successfull yet
Reason: OAuthVersionRoute_WaitingForRoute
Status: True
Type: Progressing
Last Transition Time: 2020-08-26T18:18:09Z
Message: OAuthRouteCheckEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
OAuthVersionRouteAvailable: HTTP request to "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz" failed: dial tcp: lookup oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:54946->172.30.0.10:53: i/o timeout
APIServicesAvailable: "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
APIServicesAvailable: "user.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
Reason: APIServices_Error::OAuthRouteCheckEndpointAccessibleController_EndpointUnavailable::OAuthVersionRoute_RequestFailed
Status: False
Type: Available
Last Transition Time: 2020-08-25T23:44:42Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: authentications
Group: config.openshift.io
Name: cluster
Resource: authentications
Group: config.openshift.io
Name: cluster
Resource: infrastructures
Group: config.openshift.io
Name: cluster
Resource: oauths
Group: route.openshift.io
Name: oauth-openshift
Namespace: openshift-authentication
Resource: routes
Group:
Name: oauth-openshift
Namespace: openshift-authentication
Resource: services
Group:
Name: openshift-config
Resource: namespaces
Group:
Name: openshift-config-managed
Resource: namespaces
Group:
Name: openshift-authentication
Resource: namespaces
Group:
Name: openshift-authentication-operator
Resource: namespaces
Group:
Name: openshift-ingress
Resource: namespaces
Group:
Name: openshift-oauth-apiserver
Resource: namespaces
Versions:
Name: operator
Version: 4.6.0-0.nightly-2020-08-26-103641
Name: oauth-openshift
Version: 4.6.0-0.nightly-2020-08-26-103641_openshift
Name: oauth-apiserver
Version: 4.6.0-0.nightly-2020-08-26-103641
Events: <none>
[root@rpattath ~]# oc describe co openshift-apiserver
Name: openshift-apiserver
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-08-25T23:35:23Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:spec:
f:status:
.:
f:extension:
Manager: cluster-version-operator
Operation: Update
Time: 2020-08-25T23:35:24Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
f:versions:
Manager: cluster-openshift-apiserver-operator
Operation: Update
Time: 2020-08-27T13:13:36Z
Resource Version: 1439287
Self Link: /apis/config.openshift.io/v1/clusteroperators/openshift-apiserver
UID: d2cfc538-85ae-4293-aa70-8bc96bfaf295
Spec:
Status:
Conditions:
Last Transition Time: 2020-08-26T18:18:31Z
Message: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver
Reason: APIServerDeployment_UnavailablePod
Status: True
Type: Degraded
Last Transition Time: 2020-08-26T17:58:51Z
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2020-08-27T12:41:16Z
Message: APIServicesAvailable: "build.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
APIServicesAvailable: "quota.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
Reason: APIServices_Error
Status: False
Type: Available
Last Transition Time: 2020-08-25T23:44:42Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: openshiftapiservers
Group:
Name: openshift-config
Resource: namespaces
Group:
Name: openshift-config-managed
Resource: namespaces
Group:
Name: openshift-apiserver-operator
Resource: namespaces
Group:
Name: openshift-apiserver
Resource: namespaces
Group:
Name: openshift-etcd-operator
Resource: namespaces
Group:
Name: host-etcd-2
Namespace: openshift-etcd
Resource: endpoints
Group: controlplane.operator.openshift.io
Name:
Namespace: openshift-apiserver
Resource: podnetworkconnectivitychecks
Group: apiregistration.k8s.io
Name: v1.apps.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.authorization.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.build.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.image.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.oauth.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.project.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.quota.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.route.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.security.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.template.openshift.io
Resource: apiservices
Group: apiregistration.k8s.io
Name: v1.user.openshift.io
Resource: apiservices
Versions:
Name: operator
Version: 4.6.0-0.nightly-2020-08-26-103641
Name: openshift-apiserver
Version: 4.6.0-0.nightly-2020-08-26-103641
Events: <none>
[root@rpattath ~]# oc describe co openshift-controller-manager
Name: openshift-controller-manager
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-08-25T23:35:24Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:spec:
f:status:
.:
f:extension:
Manager: cluster-version-operator
Operation: Update
Time: 2020-08-25T23:35:24Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
f:versions:
Manager: cluster-openshift-controller-manager-operator
Operation: Update
Time: 2020-08-27T13:13:53Z
Resource Version: 1439567
Self Link: /apis/config.openshift.io/v1/clusteroperators/openshift-controller-manager
UID: 6b8ae277-3676-4755-be33-7bf47f9c1025
Spec:
Status:
Conditions:
Last Transition Time: 2020-08-25T23:44:41Z
Reason: AsExpected
Status: False
Type: Degraded
Last Transition Time: 2020-08-26T18:43:53Z
Message: Progressing: daemonset/controller-manager: observed generation is 13, desired generation is 19.
Progressing: daemonset/controller-manager: number available is 0, desired number available > 1
Progressing: daemonset/controller-manager: updated number scheduled is 0, desired number scheduled is 3
Reason: _DesiredStateNotYetAchieved
Status: True
Type: Progressing
Last Transition Time: 2020-08-26T18:44:33Z
Message: Available: no daemon pods available on any node.
Reason: _NoPodsAvailable
Status: False
Type: Available
Last Transition Time: 2020-08-25T23:44:41Z
Reason: NoData
Status: Unknown
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: openshiftcontrollermanagers
Group:
Name: openshift-config
Resource: namespaces
Group:
Name: openshift-config-managed
Resource: namespaces
Group:
Name: openshift-controller-manager-operator
Resource: namespaces
Group:
Name: openshift-controller-manager
Resource: namespaces
Versions:
Name: operator
Version: 4.6.0-0.nightly-2020-08-26-103641
Events: <none>
[root@rpattath ~]# oc get ^C
[root@rpattath ~]# oc describe co machine-config
Name: machine-config
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-08-25T23:35:24Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:spec:
f:status:
.:
f:relatedObjects:
Manager: cluster-version-operator
Operation: Update
Time: 2020-08-25T23:35:24Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:extension:
.:
f:master:
f:worker:
f:versions:
Manager: machine-config-operator
Operation: Update
Time: 2020-08-26T18:18:33Z
Resource Version: 372463
Self Link: /apis/config.openshift.io/v1/clusteroperators/machine-config
UID: c15c66ae-ddbb-4993-ab0e-5b43f6937979
Spec:
Status:
Conditions:
Last Transition Time: 2020-08-26T18:16:06Z
Message: Working towards 4.6.0-0.nightly-2020-08-26-103641
Status: True
Type: Progressing
Last Transition Time: 2020-08-26T18:16:07Z
Status: False
Type: Degraded
Last Transition Time: 2020-08-26T18:14:03Z
Message: Cluster not available for 4.5.7
Status: False
Type: Available
Last Transition Time: 2020-08-25T23:45:43Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension:
Master: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-2b7eb591b2a63189bce1de2ee9683edd
Worker: all 3 nodes are at latest configuration rendered-worker-7c88f1b5eb8469ce3dde6943e04f87af
Related Objects:
Group:
Name: openshift-machine-config-operator
Resource: namespaces
Group: machineconfiguration.openshift.io
Name: master
Resource: machineconfigpools
Group: machineconfiguration.openshift.io
Name: worker
Resource: machineconfigpools
Group: machineconfiguration.openshift.io
Name: machine-config-controller
Resource: controllerconfigs
Versions:
Name: operator
Version: 4.5.7
Events: <none>
[root@rpattath ~]# oc describe co console
Name: console
Namespace:
Labels: <none>
Annotations: <none>
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-08-25T23:35:24Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:status:
.:
f:extension:
Manager: cluster-version-operator
Operation: Update
Time: 2020-08-25T23:35:24Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
f:versions:
Manager: console
Operation: Update
Time: 2020-08-27T13:20:55Z
Resource Version: 1445659
Self Link: /apis/config.openshift.io/v1/clusteroperators/console
UID: 38007251-13e4-42b9-91d6-ef72f827039f
Spec:
Status:
Conditions:
Last Transition Time: 2020-08-26T18:39:46Z
Message: CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io console-custom)
SyncLoopRefreshDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console)
RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/health): Get "https://console-openshift-console.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Reason: CustomRouteSync_FailedDeleteCustomRoutes::RouteHealth_FailedGet::SyncLoopRefresh_InProgress
Status: True
Type: Degraded
Last Transition Time: 2020-08-27T13:20:23Z
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2020-08-26T18:16:23Z
Message: DeploymentAvailable: 0 pods available for console deployment
Reason: Deployment_InsufficientReplicas
Status: False
Type: Available
Last Transition Time: 2020-08-25T23:50:56Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: consoles
Group: config.openshift.io
Name: cluster
Resource: consoles
Group: config.openshift.io
Name: cluster
Resource: infrastructures
Group: config.openshift.io
Name: cluster
Resource: proxies
Group: config.openshift.io
Name: cluster
Resource: oauths
Group: oauth.openshift.io
Name: console
Resource: oauthclients
Group:
Name: openshift-console-operator
Resource: namespaces
Group:
Name: openshift-console
Resource: namespaces
Group:
Name: console-public
Namespace: openshift-config-managed
Resource: configmaps
Versions:
Name: operator
Version: 4.6.0-0.nightly-2020-08-26-103641
Events: <none>
Moving this to auth as that seems to be where this is failing in both cases posts 3 and 5.. The error messages all point to networking. This one is especially concerning: DNS cannot be contacted OAuthVersionRouteAvailable: HTTP request to "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz" failed: dial tcp: lookup oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:54946->172.30.0.10:53: i/o timeout Also the ingress route is done, openshift-apiserver cannot be contacted through service network. Without network other components have a hard time. Failing DNS is a clear sign for it. Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1]. If you feel like this bug still needs to be a suspect, please add keyword again. [1]: https://github.com/openshift/enhancements/pull/475 |
Description of problem: OVN cluster,upgrading from 4.5.5 to 4.6 latest nightly build failed,stuck in machineconfig Version-Release number of selected component (if applicable): Base version:4.5.6 Target version: 4.6.0-0.nightly-2020-08-21-011653 How reproducible: Not sure Steps to Reproduce: 1. Install 4.5.6 baremetal OVN cluster. 2. Upgrade to latest 4.6 nightly build with below commands: oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-08-21-011653 --force=true --allow-explicit-upgrade=true Actual results: The upgrading stuck in machineconfig. oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.6.0-0.nightly-2020-08-21-011653 False True True 84m cloud-credential 4.6.0-0.nightly-2020-08-21-011653 True False False 142m cluster-autoscaler 4.6.0-0.nightly-2020-08-21-011653 True False False 132m config-operator 4.6.0-0.nightly-2020-08-21-011653 True False False 132m console 4.6.0-0.nightly-2020-08-21-011653 False False True 68m csi-snapshot-controller 4.6.0-0.nightly-2020-08-21-011653 True False False 77m dns 4.6.0-0.nightly-2020-08-21-011653 True False False 138m etcd 4.6.0-0.nightly-2020-08-21-011653 True False False 137m image-registry 4.6.0-0.nightly-2020-08-21-011653 True False False 129m ingress 4.6.0-0.nightly-2020-08-21-011653 True False False 94m insights 4.6.0-0.nightly-2020-08-21-011653 True False False 133m kube-apiserver 4.6.0-0.nightly-2020-08-21-011653 True False False 137m kube-controller-manager 4.6.0-0.nightly-2020-08-21-011653 True False False 137m kube-scheduler 4.6.0-0.nightly-2020-08-21-011653 True False False 136m kube-storage-version-migrator 4.6.0-0.nightly-2020-08-21-011653 True False False 129m machine-api 4.6.0-0.nightly-2020-08-21-011653 True False False 133m machine-approver 4.6.0-0.nightly-2020-08-21-011653 True False False 133m machine-config 4.5.6 False True True 83m marketplace 4.6.0-0.nightly-2020-08-21-011653 True False False 71m monitoring 4.6.0-0.nightly-2020-08-21-011653 False True True 78m network 4.6.0-0.nightly-2020-08-21-011653 True False False 139m node-tuning 4.6.0-0.nightly-2020-08-21-011653 True False False 93m openshift-apiserver 4.6.0-0.nightly-2020-08-21-011653 False False True 83m openshift-controller-manager 4.6.0-0.nightly-2020-08-21-011653 True False False 92m openshift-samples 4.6.0-0.nightly-2020-08-21-011653 True False False 93m operator-lifecycle-manager 4.6.0-0.nightly-2020-08-21-011653 True False False 139m operator-lifecycle-manager-catalog 4.6.0-0.nightly-2020-08-21-011653 True False False 138m operator-lifecycle-manager-packageserver 4.6.0-0.nightly-2020-08-21-011653 False True False 83m service-ca 4.6.0-0.nightly-2020-08-21-011653 True False False 139m storage 4.6.0-0.nightly-2020-08-21-011653 True False False 94m oc get co machine-config -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: annotations: exclude.release.openshift.io/internal-openshift-hosted: "true" creationTimestamp: "2020-08-21T05:55:45Z" generation: 1 managedFields: - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:exclude.release.openshift.io/internal-openshift-hosted: {} f:spec: {} f:status: {} manager: cluster-version-operator operation: Update time: "2020-08-21T05:55:45Z" - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: {} f:extension: .: {} f:lastSyncError: {} f:master: {} f:worker: {} f:relatedObjects: {} f:versions: {} manager: machine-config-operator operation: Update time: "2020-08-21T07:49:42Z" name: machine-config resourceVersion: "90155" selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-config uid: 7b59680a-efe4-400a-937e-237bc29de2cf spec: {} status: conditions: - lastTransitionTime: "2020-08-21T06:57:05Z" message: Working towards 4.6.0-0.nightly-2020-08-21-011653 status: "True" type: Progressing - lastTransitionTime: "2020-08-21T07:19:17Z" message: 'Unable to apply 4.6.0-0.nightly-2020-08-21-011653: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4aec060c0ccf9f40602603a554af0db3 expected 9a6ef99e2dbac4c6c00ba890894ed003390f5c0a has 6b77b94f2ca25d6619ca2c686232b920039c4684: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d8c3a5b98611e77bd635414f1dcb023f, retrying' reason: RequiredPoolsFailed status: "True" type: Degraded - lastTransitionTime: "2020-08-21T06:55:04Z" message: Cluster not available for 4.6.0-0.nightly-2020-08-21-011653 status: "False" type: Available - lastTransitionTime: "2020-08-21T06:00:51Z" reason: AsExpected status: "True" type: Upgradeable extension: lastSyncError: 'pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4aec060c0ccf9f40602603a554af0db3 expected 9a6ef99e2dbac4c6c00ba890894ed003390f5c0a has 6b77b94f2ca25d6619ca2c686232b920039c4684: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d8c3a5b98611e77bd635414f1dcb023f, retrying' master: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d8c3a5b98611e77bd635414f1dcb023f worker: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-worker-2731753cc091ab217efc740a0dda27cb relatedObjects: - group: "" name: openshift-machine-config-operator resource: namespaces - group: machineconfiguration.openshift.io name: "" resource: machineconfigpools - group: machineconfiguration.openshift.io name: "" resource: controllerconfigs - group: machineconfiguration.openshift.io name: "" resource: kubeletconfigs - group: machineconfiguration.openshift.io name: "" resource: containerruntimeconfigs - group: machineconfiguration.openshift.io name: "" resource: machineconfigs versions: - name: operator version: 4.5.6 huiran-mac:script hrwang$ oc get nodes NAME STATUS ROLES AGE VERSION huir-456-m26z7-compute-0 Ready worker 113m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-compute-1 Ready worker 113m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-compute-2 Ready worker 113m v1.18.3+002a51f huir-456-m26z7-control-plane-0 Ready master 123m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-control-plane-1 Ready master 124m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-control-plane-2 Ready master 123m v1.18.3+002a51f Expected results: Upgrade successfully. Additional info: