Description of problem: OVN cluster,upgrading from 4.5.5 to 4.6 latest nightly build failed,stuck in machineconfig Version-Release number of selected component (if applicable): Base version:4.5.6 Target version: 4.6.0-0.nightly-2020-08-21-011653 How reproducible: Not sure Steps to Reproduce: 1. Install 4.5.6 baremetal OVN cluster. 2. Upgrade to latest 4.6 nightly build with below commands: oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-08-21-011653 --force=true --allow-explicit-upgrade=true Actual results: The upgrading stuck in machineconfig. oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.6.0-0.nightly-2020-08-21-011653 False True True 84m cloud-credential 4.6.0-0.nightly-2020-08-21-011653 True False False 142m cluster-autoscaler 4.6.0-0.nightly-2020-08-21-011653 True False False 132m config-operator 4.6.0-0.nightly-2020-08-21-011653 True False False 132m console 4.6.0-0.nightly-2020-08-21-011653 False False True 68m csi-snapshot-controller 4.6.0-0.nightly-2020-08-21-011653 True False False 77m dns 4.6.0-0.nightly-2020-08-21-011653 True False False 138m etcd 4.6.0-0.nightly-2020-08-21-011653 True False False 137m image-registry 4.6.0-0.nightly-2020-08-21-011653 True False False 129m ingress 4.6.0-0.nightly-2020-08-21-011653 True False False 94m insights 4.6.0-0.nightly-2020-08-21-011653 True False False 133m kube-apiserver 4.6.0-0.nightly-2020-08-21-011653 True False False 137m kube-controller-manager 4.6.0-0.nightly-2020-08-21-011653 True False False 137m kube-scheduler 4.6.0-0.nightly-2020-08-21-011653 True False False 136m kube-storage-version-migrator 4.6.0-0.nightly-2020-08-21-011653 True False False 129m machine-api 4.6.0-0.nightly-2020-08-21-011653 True False False 133m machine-approver 4.6.0-0.nightly-2020-08-21-011653 True False False 133m machine-config 4.5.6 False True True 83m marketplace 4.6.0-0.nightly-2020-08-21-011653 True False False 71m monitoring 4.6.0-0.nightly-2020-08-21-011653 False True True 78m network 4.6.0-0.nightly-2020-08-21-011653 True False False 139m node-tuning 4.6.0-0.nightly-2020-08-21-011653 True False False 93m openshift-apiserver 4.6.0-0.nightly-2020-08-21-011653 False False True 83m openshift-controller-manager 4.6.0-0.nightly-2020-08-21-011653 True False False 92m openshift-samples 4.6.0-0.nightly-2020-08-21-011653 True False False 93m operator-lifecycle-manager 4.6.0-0.nightly-2020-08-21-011653 True False False 139m operator-lifecycle-manager-catalog 4.6.0-0.nightly-2020-08-21-011653 True False False 138m operator-lifecycle-manager-packageserver 4.6.0-0.nightly-2020-08-21-011653 False True False 83m service-ca 4.6.0-0.nightly-2020-08-21-011653 True False False 139m storage 4.6.0-0.nightly-2020-08-21-011653 True False False 94m oc get co machine-config -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: annotations: exclude.release.openshift.io/internal-openshift-hosted: "true" creationTimestamp: "2020-08-21T05:55:45Z" generation: 1 managedFields: - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:exclude.release.openshift.io/internal-openshift-hosted: {} f:spec: {} f:status: {} manager: cluster-version-operator operation: Update time: "2020-08-21T05:55:45Z" - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: {} f:extension: .: {} f:lastSyncError: {} f:master: {} f:worker: {} f:relatedObjects: {} f:versions: {} manager: machine-config-operator operation: Update time: "2020-08-21T07:49:42Z" name: machine-config resourceVersion: "90155" selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-config uid: 7b59680a-efe4-400a-937e-237bc29de2cf spec: {} status: conditions: - lastTransitionTime: "2020-08-21T06:57:05Z" message: Working towards 4.6.0-0.nightly-2020-08-21-011653 status: "True" type: Progressing - lastTransitionTime: "2020-08-21T07:19:17Z" message: 'Unable to apply 4.6.0-0.nightly-2020-08-21-011653: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4aec060c0ccf9f40602603a554af0db3 expected 9a6ef99e2dbac4c6c00ba890894ed003390f5c0a has 6b77b94f2ca25d6619ca2c686232b920039c4684: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d8c3a5b98611e77bd635414f1dcb023f, retrying' reason: RequiredPoolsFailed status: "True" type: Degraded - lastTransitionTime: "2020-08-21T06:55:04Z" message: Cluster not available for 4.6.0-0.nightly-2020-08-21-011653 status: "False" type: Available - lastTransitionTime: "2020-08-21T06:00:51Z" reason: AsExpected status: "True" type: Upgradeable extension: lastSyncError: 'pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4aec060c0ccf9f40602603a554af0db3 expected 9a6ef99e2dbac4c6c00ba890894ed003390f5c0a has 6b77b94f2ca25d6619ca2c686232b920039c4684: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d8c3a5b98611e77bd635414f1dcb023f, retrying' master: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d8c3a5b98611e77bd635414f1dcb023f worker: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-worker-2731753cc091ab217efc740a0dda27cb relatedObjects: - group: "" name: openshift-machine-config-operator resource: namespaces - group: machineconfiguration.openshift.io name: "" resource: machineconfigpools - group: machineconfiguration.openshift.io name: "" resource: controllerconfigs - group: machineconfiguration.openshift.io name: "" resource: kubeletconfigs - group: machineconfiguration.openshift.io name: "" resource: containerruntimeconfigs - group: machineconfiguration.openshift.io name: "" resource: machineconfigs versions: - name: operator version: 4.5.6 huiran-mac:script hrwang$ oc get nodes NAME STATUS ROLES AGE VERSION huir-456-m26z7-compute-0 Ready worker 113m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-compute-1 Ready worker 113m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-compute-2 Ready worker 113m v1.18.3+002a51f huir-456-m26z7-control-plane-0 Ready master 123m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-control-plane-1 Ready master 124m v1.19.0-rc.2+99cb93a-dirty huir-456-m26z7-control-plane-2 Ready master 123m v1.18.3+002a51f Expected results: Upgrade successfully. Additional info:
Seeing the same during 4.5.7 to 4.6 nightly upgrade. Marking upgrade blocker. # oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.6.0-0.nightly-2020-08-26-103641 False True True 18h cloud-credential 4.6.0-0.nightly-2020-08-26-103641 True False False 36h cluster-autoscaler 4.6.0-0.nightly-2020-08-26-103641 True False False 36h config-operator 4.6.0-0.nightly-2020-08-26-103641 True False False 36h console 4.6.0-0.nightly-2020-08-26-103641 False False True 18h csi-snapshot-controller 4.6.0-0.nightly-2020-08-26-103641 True False False 18h dns 4.6.0-0.nightly-2020-08-26-103641 True False True 18h etcd 4.6.0-0.nightly-2020-08-26-103641 True False False 36h image-registry 4.6.0-0.nightly-2020-08-26-103641 True False False 36h ingress 4.6.0-0.nightly-2020-08-26-103641 True False False 18h insights 4.6.0-0.nightly-2020-08-26-103641 True False True 36h kube-apiserver 4.6.0-0.nightly-2020-08-26-103641 True False False 36h kube-controller-manager 4.6.0-0.nightly-2020-08-26-103641 True False False 36h kube-scheduler 4.6.0-0.nightly-2020-08-26-103641 True False False 36h kube-storage-version-migrator 4.6.0-0.nightly-2020-08-26-103641 True False False 36h machine-api 4.6.0-0.nightly-2020-08-26-103641 True False False 36h machine-approver 4.6.0-0.nightly-2020-08-26-103641 True False False 36h machine-config 4.5.7 False True False 18h marketplace 4.6.0-0.nightly-2020-08-26-103641 False False False 18h monitoring 4.6.0-0.nightly-2020-08-26-103641 False True True 17h network 4.6.0-0.nightly-2020-08-26-103641 True False False 36h node-tuning 4.6.0-0.nightly-2020-08-26-103641 True False False 18h openshift-apiserver 4.6.0-0.nightly-2020-08-26-103641 False False True 138m openshift-controller-manager 4.6.0-0.nightly-2020-08-26-103641 False True False 17h openshift-samples 4.6.0-0.nightly-2020-08-26-103641 True False False 17h operator-lifecycle-manager 4.6.0-0.nightly-2020-08-26-103641 True False False 36h operator-lifecycle-manager-catalog 4.6.0-0.nightly-2020-08-26-103641 True False False 36h operator-lifecycle-manager-packageserver 4.6.0-0.nightly-2020-08-26-103641 True False False 18h service-ca 4.6.0-0.nightly-2020-08-26-103641 True False False 36h storage 4.6.0-0.nightly-2020-08-26-103641 True False False 18h [root@rpattath ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.7 True True 18h Unable to apply 4.6.0-0.nightly-2020-08-26-103641: the control plane is reporting an internal error [root@rpattath ~]# oc get pods --all-namespaces | egrep -v "Running|Complete" NAMESPACE NAME READY STATUS RESTARTS AGE openshift-etcd etcd-quorum-guard-647745d8c-sjjmr 0/1 Pending 0 18h [root@rpattath ~]# oc logs pod/etcd-quorum-guard-647745d8c-sjjmr -n openshift-etcd [root@rpattath ~]# oc describe co authentication Name: authentication Namespace: Labels: <none> Annotations: exclude.release.openshift.io/internal-openshift-hosted: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-08-25T23:35:24Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:exclude.release.openshift.io/internal-openshift-hosted: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2020-08-25T23:35:24Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:relatedObjects: f:versions: Manager: authentication-operator Operation: Update Time: 2020-08-27T13:13:12Z Resource Version: 1438927 Self Link: /apis/config.openshift.io/v1/clusteroperators/authentication UID: 2a6a2718-474c-4429-8c5a-3a55126aa10e Spec: Status: Conditions: Last Transition Time: 2020-08-26T18:18:30Z Message: OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) OAuthServiceEndpointsCheckEndpointAccessibleControllerDegraded: Get "https://10.130.0.31:6443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) OAuthServerDeploymentDegraded: Unable to get "openshift-browser-client" bootstrapped OAuth client: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io openshift-browser-client) APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver Reason: APIServerDeployment_UnavailablePod::OAuthRouteCheckEndpointAccessibleController_SyncError::OAuthServerDeployment_GetFailed::OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError Status: True Type: Degraded Last Transition Time: 2020-08-26T18:18:29Z Message: OAuthVersionRouteProgressing: Request to "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz" not successfull yet Reason: OAuthVersionRoute_WaitingForRoute Status: True Type: Progressing Last Transition Time: 2020-08-26T18:18:09Z Message: OAuthRouteCheckEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) OAuthVersionRouteAvailable: HTTP request to "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz" failed: dial tcp: lookup oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:54946->172.30.0.10:53: i/o timeout APIServicesAvailable: "oauth.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "user.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) Reason: APIServices_Error::OAuthRouteCheckEndpointAccessibleController_EndpointUnavailable::OAuthVersionRoute_RequestFailed Status: False Type: Available Last Transition Time: 2020-08-25T23:44:42Z Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: authentications Group: config.openshift.io Name: cluster Resource: authentications Group: config.openshift.io Name: cluster Resource: infrastructures Group: config.openshift.io Name: cluster Resource: oauths Group: route.openshift.io Name: oauth-openshift Namespace: openshift-authentication Resource: routes Group: Name: oauth-openshift Namespace: openshift-authentication Resource: services Group: Name: openshift-config Resource: namespaces Group: Name: openshift-config-managed Resource: namespaces Group: Name: openshift-authentication Resource: namespaces Group: Name: openshift-authentication-operator Resource: namespaces Group: Name: openshift-ingress Resource: namespaces Group: Name: openshift-oauth-apiserver Resource: namespaces Versions: Name: operator Version: 4.6.0-0.nightly-2020-08-26-103641 Name: oauth-openshift Version: 4.6.0-0.nightly-2020-08-26-103641_openshift Name: oauth-apiserver Version: 4.6.0-0.nightly-2020-08-26-103641 Events: <none> [root@rpattath ~]# oc describe co openshift-apiserver Name: openshift-apiserver Namespace: Labels: <none> Annotations: exclude.release.openshift.io/internal-openshift-hosted: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-08-25T23:35:23Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:exclude.release.openshift.io/internal-openshift-hosted: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2020-08-25T23:35:24Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:relatedObjects: f:versions: Manager: cluster-openshift-apiserver-operator Operation: Update Time: 2020-08-27T13:13:36Z Resource Version: 1439287 Self Link: /apis/config.openshift.io/v1/clusteroperators/openshift-apiserver UID: d2cfc538-85ae-4293-aa70-8bc96bfaf295 Spec: Status: Conditions: Last Transition Time: 2020-08-26T18:18:31Z Message: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver Reason: APIServerDeployment_UnavailablePod Status: True Type: Degraded Last Transition Time: 2020-08-26T17:58:51Z Reason: AsExpected Status: False Type: Progressing Last Transition Time: 2020-08-27T12:41:16Z Message: APIServicesAvailable: "build.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) APIServicesAvailable: "quota.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request) Reason: APIServices_Error Status: False Type: Available Last Transition Time: 2020-08-25T23:44:42Z Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: openshiftapiservers Group: Name: openshift-config Resource: namespaces Group: Name: openshift-config-managed Resource: namespaces Group: Name: openshift-apiserver-operator Resource: namespaces Group: Name: openshift-apiserver Resource: namespaces Group: Name: openshift-etcd-operator Resource: namespaces Group: Name: host-etcd-2 Namespace: openshift-etcd Resource: endpoints Group: controlplane.operator.openshift.io Name: Namespace: openshift-apiserver Resource: podnetworkconnectivitychecks Group: apiregistration.k8s.io Name: v1.apps.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.authorization.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.build.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.image.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.oauth.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.project.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.quota.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.route.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.security.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.template.openshift.io Resource: apiservices Group: apiregistration.k8s.io Name: v1.user.openshift.io Resource: apiservices Versions: Name: operator Version: 4.6.0-0.nightly-2020-08-26-103641 Name: openshift-apiserver Version: 4.6.0-0.nightly-2020-08-26-103641 Events: <none> [root@rpattath ~]# oc describe co openshift-controller-manager Name: openshift-controller-manager Namespace: Labels: <none> Annotations: exclude.release.openshift.io/internal-openshift-hosted: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-08-25T23:35:24Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:exclude.release.openshift.io/internal-openshift-hosted: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2020-08-25T23:35:24Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:relatedObjects: f:versions: Manager: cluster-openshift-controller-manager-operator Operation: Update Time: 2020-08-27T13:13:53Z Resource Version: 1439567 Self Link: /apis/config.openshift.io/v1/clusteroperators/openshift-controller-manager UID: 6b8ae277-3676-4755-be33-7bf47f9c1025 Spec: Status: Conditions: Last Transition Time: 2020-08-25T23:44:41Z Reason: AsExpected Status: False Type: Degraded Last Transition Time: 2020-08-26T18:43:53Z Message: Progressing: daemonset/controller-manager: observed generation is 13, desired generation is 19. Progressing: daemonset/controller-manager: number available is 0, desired number available > 1 Progressing: daemonset/controller-manager: updated number scheduled is 0, desired number scheduled is 3 Reason: _DesiredStateNotYetAchieved Status: True Type: Progressing Last Transition Time: 2020-08-26T18:44:33Z Message: Available: no daemon pods available on any node. Reason: _NoPodsAvailable Status: False Type: Available Last Transition Time: 2020-08-25T23:44:41Z Reason: NoData Status: Unknown Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: openshiftcontrollermanagers Group: Name: openshift-config Resource: namespaces Group: Name: openshift-config-managed Resource: namespaces Group: Name: openshift-controller-manager-operator Resource: namespaces Group: Name: openshift-controller-manager Resource: namespaces Versions: Name: operator Version: 4.6.0-0.nightly-2020-08-26-103641 Events: <none> [root@rpattath ~]# oc get ^C [root@rpattath ~]# oc describe co machine-config Name: machine-config Namespace: Labels: <none> Annotations: exclude.release.openshift.io/internal-openshift-hosted: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-08-25T23:35:24Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:exclude.release.openshift.io/internal-openshift-hosted: f:spec: f:status: .: f:relatedObjects: Manager: cluster-version-operator Operation: Update Time: 2020-08-25T23:35:24Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:extension: .: f:master: f:worker: f:versions: Manager: machine-config-operator Operation: Update Time: 2020-08-26T18:18:33Z Resource Version: 372463 Self Link: /apis/config.openshift.io/v1/clusteroperators/machine-config UID: c15c66ae-ddbb-4993-ab0e-5b43f6937979 Spec: Status: Conditions: Last Transition Time: 2020-08-26T18:16:06Z Message: Working towards 4.6.0-0.nightly-2020-08-26-103641 Status: True Type: Progressing Last Transition Time: 2020-08-26T18:16:07Z Status: False Type: Degraded Last Transition Time: 2020-08-26T18:14:03Z Message: Cluster not available for 4.5.7 Status: False Type: Available Last Transition Time: 2020-08-25T23:45:43Z Reason: AsExpected Status: True Type: Upgradeable Extension: Master: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-2b7eb591b2a63189bce1de2ee9683edd Worker: all 3 nodes are at latest configuration rendered-worker-7c88f1b5eb8469ce3dde6943e04f87af Related Objects: Group: Name: openshift-machine-config-operator Resource: namespaces Group: machineconfiguration.openshift.io Name: master Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: worker Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: machine-config-controller Resource: controllerconfigs Versions: Name: operator Version: 4.5.7 Events: <none> [root@rpattath ~]# oc describe co console Name: console Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-08-25T23:35:24Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2020-08-25T23:35:24Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:relatedObjects: f:versions: Manager: console Operation: Update Time: 2020-08-27T13:20:55Z Resource Version: 1445659 Self Link: /apis/config.openshift.io/v1/clusteroperators/console UID: 38007251-13e4-42b9-91d6-ef72f827039f Spec: Status: Conditions: Last Transition Time: 2020-08-26T18:39:46Z Message: CustomRouteSyncDegraded: the server is currently unable to handle the request (delete routes.route.openshift.io console-custom) SyncLoopRefreshDegraded: the server is currently unable to handle the request (get routes.route.openshift.io console) RouteHealthDegraded: failed to GET route (https://console-openshift-console.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/health): Get "https://console-openshift-console.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Reason: CustomRouteSync_FailedDeleteCustomRoutes::RouteHealth_FailedGet::SyncLoopRefresh_InProgress Status: True Type: Degraded Last Transition Time: 2020-08-27T13:20:23Z Reason: AsExpected Status: False Type: Progressing Last Transition Time: 2020-08-26T18:16:23Z Message: DeploymentAvailable: 0 pods available for console deployment Reason: Deployment_InsufficientReplicas Status: False Type: Available Last Transition Time: 2020-08-25T23:50:56Z Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: consoles Group: config.openshift.io Name: cluster Resource: consoles Group: config.openshift.io Name: cluster Resource: infrastructures Group: config.openshift.io Name: cluster Resource: proxies Group: config.openshift.io Name: cluster Resource: oauths Group: oauth.openshift.io Name: console Resource: oauthclients Group: Name: openshift-console-operator Resource: namespaces Group: Name: openshift-console Resource: namespaces Group: Name: console-public Namespace: openshift-config-managed Resource: configmaps Versions: Name: operator Version: 4.6.0-0.nightly-2020-08-26-103641 Events: <none>
Moving this to auth as that seems to be where this is failing in both cases posts 3 and 5..
The error messages all point to networking. This one is especially concerning: DNS cannot be contacted OAuthVersionRouteAvailable: HTTP request to "https://oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com/healthz" failed: dial tcp: lookup oauth-openshift.apps.rpattath-45-ovnupgrade.qe.devcluster.openshift.com on 172.30.0.10:53: read udp 10.128.0.8:54946->172.30.0.10:53: i/o timeout Also the ingress route is done, openshift-apiserver cannot be contacted through service network. Without network other components have a hard time. Failing DNS is a clear sign for it.
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1]. If you feel like this bug still needs to be a suspect, please add keyword again. [1]: https://github.com/openshift/enhancements/pull/475