Please run must-gather and attach the output, if that's possible in that cluster state. CVO logs would be most interesting here.
Hello Stefan, This is a disconnected environment with an offline registry. Without Internet access on that server the must-gather would fail (opened https://bugzilla.redhat.com/show_bug.cgi?id=1823839 for this): ~~~ $ oc adm must-gather [must-gather ] OUT unable to resolve the imagestream tag openshift/must-gather:latest [must-gather ] OUT [must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest [must-gather ] OUT namespace/openshift-must-gather-t7zlg created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-gwpdf created [must-gather ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created [must-gather-ldn97] OUT gather did not start: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 18.208.7.119:443: i/o timeout [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-gwpdf deleted [must-gather ] OUT namespace/openshift-must-gather-t7zlg deleted error: gather did not start for pod must-gather-ldn97: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 18.208.7.119:443: i/o timeout ~~~ I was able to enable the Internet on this cluster to gather the information, but this might not fully preserve the cluster state. The cluster operators still show the same status: $ ./oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.3.0-0.nightly-2020-04-13-190424 True False False 8h cloud-credential 4.3.0-0.nightly-2020-04-13-190424 True False False 9h cluster-autoscaler 4.3.0-0.nightly-2020-04-13-190424 True False False 9h console 4.3.0-0.nightly-2020-04-13-190424 True False False 8h dns 4.3.0-0.nightly-2020-04-13-190424 True False False 9h image-registry 4.3.0-0.nightly-2020-04-13-190424 True False False 8h ingress 4.3.0-0.nightly-2020-04-13-190424 True False False 8h insights 4.3.0-0.nightly-2020-04-13-190424 True False False 9h kube-apiserver 4.3.0-0.nightly-2020-04-13-190424 True False False 9h kube-controller-manager 4.3.0-0.nightly-2020-04-13-190424 True False False 9h kube-scheduler 4.3.0-0.nightly-2020-04-13-190424 True False False 9h machine-api 4.3.0-0.nightly-2020-04-13-190424 True False False 9h machine-config 4.3.0-0.nightly-2020-04-13-190424 True False False 9h marketplace 4.3.0-0.nightly-2020-04-13-190424 True False False 9h monitoring 4.3.0-0.nightly-2020-04-13-190424 False True True 34m network 4.3.0-0.nightly-2020-04-13-190424 True False False 9h node-tuning 4.3.0-0.nightly-2020-04-13-190424 True False False 9h openshift-apiserver 4.3.0-0.nightly-2020-04-13-190424 False False False 65m openshift-controller-manager 4.3.0-0.nightly-2020-04-13-190424 True False False 9h openshift-samples 4.3.0-0.nightly-2020-04-13-190424 True False False 8h operator-lifecycle-manager 4.3.0-0.nightly-2020-04-13-190424 True False False 9h operator-lifecycle-manager-catalog 4.3.0-0.nightly-2020-04-13-190424 True False False 9h operator-lifecycle-manager-packageserver 4.3.0-0.nightly-2020-04-13-190424 True False False 3h4m service-ca 4.3.0-0.nightly-2020-04-13-190424 True False False 9h service-catalog-apiserver 4.3.0-0.nightly-2020-04-13-190424 True False False 9h service-catalog-controller-manager 4.3.0-0.nightly-2020-04-13-190424 True False False 9h storage 4.3.0-0.nightly-2020-04-13-190424 True False False 9h
Unable to reproduce this bug for UPI install on azure, close it. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.13 True False 67m Cluster version is 4.3.13 $ oc get nodes NAME STATUS ROLES AGE VERSION keazure4313-04170145-master-0 Ready master 98m v1.16.2 keazure4313-04170145-master-1 Ready master 98m v1.16.2 keazure4313-04170145-master-2 Ready master 98m v1.16.2 keazure4313-04170145-worker-centralus-1 Ready worker 79m v1.16.2 keazure4313-04170145-worker-centralus-2 Ready worker 79m v1.16.2 keazure4313-04170145-worker-centralus-3 Ready worker 79m v1.16.2 $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.3.13 True False False 70m cloud-credential 4.3.13 True False False 99m cluster-autoscaler 4.3.13 True False False 88m console 4.3.13 True False False 34m dns 4.3.13 True False False 94m image-registry 4.3.13 True False False 50m ingress 4.3.13 True False False 37m insights 4.3.13 True False False 90m kube-apiserver 4.3.13 True False False 92m kube-controller-manager 4.3.13 True False False 93m kube-scheduler 4.3.13 True False False 93m machine-api 4.3.13 True False False 95m machine-config 4.3.13 True False False 94m marketplace 4.3.13 True False False 37m monitoring 4.3.13 True False False 32m network 4.3.13 True False False 95m node-tuning 4.3.13 True False False 38m openshift-apiserver 4.3.13 True False False 32m openshift-controller-manager 4.3.13 True False False 91m openshift-samples 4.3.13 True False False 29m operator-lifecycle-manager 4.3.13 True False False 91m operator-lifecycle-manager-catalog 4.3.13 True False False 91m operator-lifecycle-manager-packageserver 4.3.13 True False False 35m service-ca 4.3.13 True False False 95m service-catalog-apiserver 4.3.13 True False False 91m service-catalog-controller-manager 4.3.13 True False False 90m storage 4.3.13 True False False 89m