Version: 4.4.5 After minor update from 4.4.4 to 4.4.5 attempted to restore to previous version from backup. Attempt to run oc commands fails with: The connection to the server api.ocp-sasha-1.qe.lab.redhat.com:6443 was refused - did you specify the right host or port? Steps to reproduce: ############################################# check things before action ################################ [kni@provisionhost-0-0 ~]$ oc get node --context spoke NAME STATUS ROLES AGE VERSION master-1-0 Ready master 118m v1.17.1 master-1-1 Ready master 118m v1.17.1 master-1-2 Ready master 119m v1.17.1 worker-1-0 Ready worker 99m v1.17.1 worker-1-1 Ready worker 98m v1.17.1 [kni@provisionhost-0-0 ~]$ oc get co --context spoke NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.4.4 True False False 91m cloud-credential 4.4.4 True False False 130m cluster-autoscaler 4.4.4 True False False 106m console 4.4.4 True False False 93m csi-snapshot-controller 4.4.4 True False False 110m dns 4.4.4 True False False 116m etcd 4.4.4 True False False 114m image-registry 4.4.4 True False False 112m ingress 4.4.4 True False False 97m insights 4.4.4 True False False 112m kube-apiserver 4.4.4 True False False 114m kube-controller-manager 4.4.4 True False False 116m kube-scheduler 4.4.4 True False False 115m kube-storage-version-migrator 4.4.4 True False False 117m machine-api 4.4.4 True False False 116m machine-config 4.4.4 True False False 114m marketplace 4.4.4 True False False 96m monitoring 4.4.4 True False False 95m network 4.4.4 True False False 117m node-tuning 4.4.4 True False False 117m openshift-apiserver 4.4.4 True False False 94m openshift-controller-manager 4.4.4 True False False 110m openshift-samples 4.4.4 True False False 96m operator-lifecycle-manager 4.4.4 True False False 116m operator-lifecycle-manager-catalog 4.4.4 True False False 116m operator-lifecycle-manager-packageserver 4.4.4 True False False 95m service-ca 4.4.4 True False False 117m service-catalog-apiserver 4.4.4 True False False 118m service-catalog-controller-manager 4.4.4 True False False 117m storage 4.4.4 True False False 112m ################################### backup ###################################### core@master-1-0 ~]$ sudo -E /usr/local/bin/cluster-backup.sh ./assets/backup 1637d3833deabec3e3a41e238f3d993f57ad9f172fec99b641bf137422a1886d etcdctl version: 3.3.18 API version: 3.3 found latest kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-8 found latest kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8 found latest kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-7 found latest etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-2 Snapshot saved at ./assets/backup/snapshot_2020-06-16_201545.db snapshot db and kube resources are successfully saved to ./assets/backup [core@master-1-0 ~]$ ls [kni@provisionhost-0-0 ~]$ oc get --context spoke clusterversion -o json|jq ".items[0].spec" { "channel": "stable-4.4", "clusterID": "1aa226c1-05f5-435d-84c8-3bfacce478c3", "upstream": "https://api.openshift.com/api/upgrades_info/v1/graph" } ########################################### UPGRADE #################################### [kni@provisionhost-0-0 ~]$ oc adm upgrade --context spoke Cluster version is 4.4.4 Updates: VERSION IMAGE 4.4.5 quay.io/openshift-release-dev/ocp-release@sha256:4a461dc23a9d323c8bd7a8631bed078a9e5eec690ce073f78b645c83fb4cdf74 4.4.6 quay.io/openshift-release-dev/ocp-release@sha256:7613d8f7db639147b91b16b54b24cfa351c3cbde6aa7b7bf1b9c80c260efad06 [kni@provisionhost-0-0 ~]$ [kni@provisionhost-0-0 ~]$ oc adm upgrade --context spoke --to=4.4.5 Updating to 4.4.5 ####################################### Check things after upgrade ######################################################### [kni@provisionhost-0-0 ~]$ oc get clusterversion --context spoke NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.5 True False 6m18s Cluster version is 4.4.5 [kni@provisionhost-0-0 ~]$ oc get co --context spoke NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.4.5 True False False 169m cloud-credential 4.4.5 True False False 3h28m cluster-autoscaler 4.4.5 True False False 3h4m console 4.4.5 True False False 15m csi-snapshot-controller 4.4.5 True False False 29m dns 4.4.5 True False False 3h14m etcd 4.4.5 True False False 3h12m image-registry 4.4.5 True False False 3h11m ingress 4.4.5 True False False 29m insights 4.4.5 True False False 3h10m kube-apiserver 4.4.5 True False False 3h12m kube-controller-manager 4.4.5 True False False 3h14m kube-scheduler 4.4.5 True False False 3h13m kube-storage-version-migrator 4.4.5 True False False 28m machine-api 4.4.5 True False False 3h15m machine-config 4.4.5 True False False 8m12s marketplace 4.4.5 True False False 27m monitoring 4.4.5 True False False 67m network 4.4.5 True False False 3h15m node-tuning 4.4.5 True False False 55m openshift-apiserver 4.4.5 True False False 15m openshift-controller-manager 4.4.5 True False False 3h8m openshift-samples 4.4.5 True False False 55m operator-lifecycle-manager 4.4.5 True False False 3h15m operator-lifecycle-manager-catalog 4.4.5 True False False 3h15m operator-lifecycle-manager-packageserver 4.4.5 True False False 15m service-ca 4.4.5 True False False 3h15m service-catalog-apiserver 4.4.5 True False False 3h16m service-catalog-controller-manager 4.4.5 True False False 3h15m storage 4.4.5 True False False 55m [kni@provisionhost-0-0 ~]$ oc get pod -A --context spoke |grep -v Run|grep -v Comple NAMESPACE NAME READY STATUS RESTARTS AGE [kni@provisionhost-0-0 ~]$ ##################################################### attempt to restore ##################################################### [kni@provisionhost-0-0 ~]$ oc get node --context spoke NAME STATUS ROLES AGE VERSION master-1-0 Ready master 4h1m v1.17.1 master-1-1 Ready master 4h1m v1.17.1 master-1-2 Ready master 4h2m v1.17.1 worker-1-0 Ready worker 3h41m v1.17.1 worker-1-1 Ready worker 3h40m v1.17.1 [kni@provisionhost-0-0 ~]$ [kni@provisionhost-0-0 ~]$ ssh core@master-1-0 ls assets/backup snapshot_2020-06-16_201545.db static_kuberesources_2020-06-16_201545.tar.gz [kni@provisionhost-0-0 ~]$ #stopping the static pods on other master hosts: [core@master-1-1 ~]$ sudo crictl ps|grep etcd 4d5958c8501d5 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 57 minutes ago Running etcd-metrics 0 87001a1d8518e a55a50e76cfb5 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 57 minutes ago Running etcd 0 87001a1d8518e 8693108a5673f 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 57 minutes ago Running etcdctl 0 87001a1d8518e [core@master-1-1 ~]$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp [core@master-1-1 ~]$ sudo crictl ps|grep etcd [core@master-1-1 ~]$ [core@master-1-1 ~]$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp [core@master-1-1 ~]$ sudo mv /var/lib/etcd/ /tmp [core@master-1-1 ~]$ [core@master-1-2 ~]$ sudo crictl ps|grep etcd 268cc75fcba59 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 About an hour ago Running etcd-metrics 0 e27e8ae80cf34 c0b4e9d73c146 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 About an hour ago Running etcd 0 e27e8ae80cf34 67bb7b38724fd 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 About an hour ago Running etcdctl 0 e27e8ae80cf34 [core@master-1-2 ~]$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp [core@master-1-2 ~]$ sudo crictl ps|grep etcd [core@master-1-2 ~]$ [core@master-1-2 ~]$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp [core@master-1-2 ~]$ sudo mv /var/lib/etcd/ /tmp [core@master-1-2 ~]$ [core@master-1-0 ~]$ sudo mv assets/backup . [core@master-1-0 ~]$ ls assets backup [core@master-1-0 ~]$ sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup ...stopping haproxy.yaml ...stopping keepalived.yaml ...stopping mdns-publisher.yaml ...stopping etcd-pod.yaml ...stopping coredns.yaml ...stopping kube-scheduler-pod.yaml ...stopping kube-controller-manager-pod.yaml ...stopping kube-apiserver-pod.yaml Waiting for container etcd to stop .complete Waiting for container etcdctl to stop .............................complete Waiting for container etcd-metrics to stop complete Waiting for container kube-controller-manager to stop complete Waiting for container kube-apiserver to stop complete Waiting for container kube-scheduler to stop complete Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup starting restore-etcd static pod starting kube-apiserver-pod.yaml static-pod-resources/kube-apiserver-pod-8/kube-apiserver-pod.yaml starting kube-controller-manager-pod.yaml static-pod-resources/kube-controller-manager-pod-8/kube-controller-manager-pod.yaml starting kube-scheduler-pod.yaml static-pod-resources/kube-scheduler-pod-7/kube-scheduler-pod.yaml [core@master-1-0 ~]$ sudo crictl ps | grep etcd 96fcaab8e3748 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 3 minutes ago Running etcd 0 4e2ea7b43b955 [core@master-1-0 ~]$ [kni@provisionhost-0-0 ~]$ oc get pod --context spoke -n openshift-etcd The connection to the server api.ocp-sasha-1.qe.lab.redhat.com:6443 was refused - did you specify the right host or port? [kni@provisionhost-0-0 ~]$
[core@master-1-0 ~]$ sudo crictl ps CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID b51be95a8e0a1 e5a0529c7e9a872cc13535dfa4f22a42ff0a1ea231b8cb441bfb647158b772a2 1 second ago Running prometheus-adapter 178 f98b78d51dbbb 0af0b5b793504 05354db57daa1b551680c37ac8c69c0ae211f8299a73b07310a51347866a67f8 11 seconds ago Running kube-storage-version-migrator-operator 163 287c94254db57 636d8b17eea3f e5a0529c7e9a872cc13535dfa4f22a42ff0a1ea231b8cb441bfb647158b772a2 22 seconds ago Running prometheus-adapter 178 de86457a7b8f9 740f80d7a1e4c 59e3c82b18ee7d374feb770d2948807a3ce49f58bd20029814f7faf8735dcb57 25 seconds ago Running kube-state-metrics 178 3e5af3fb262b7 684321ee7cd9d 34999445a8e16b0f806263f3c3c12cbc548d9126c717735867e01f831daf529e 32 seconds ago Running operator 163 680153b580e15 1bca6a7718e6b da25061cf9c9040bbaa7d48d6d450f3ec85816f3df21acdf116eb5a31ad6d3e9 About a minute ago Running console 168 a5c10fd73b403 ee52bac9e8d6b da25061cf9c9040bbaa7d48d6d450f3ec85816f3df21acdf116eb5a31ad6d3e9 2 minutes ago Running console 168 16c6706455fdb 2b8cca6af8274 2d6ec0f11ffd765e85df5b4a9a058fc0f7631139df70c95889e2a318a5068efb 4 minutes ago Running cluster-policy-controller 142 ccd97f003d5c0 c06f239e07f04 ee010c0ddb991e8fdb05bcc0f18c8289a24f8b534bf0b69796ffb6527ac42f40 9 hours ago Running config-reloader 0 1d558be747cd8 5568d3e6a884b 29255c72af1e31b5b353815da5d18e554bb03ac811569e9320e865a60c5050db 9 hours ago Running alertmanager 0 1d558be747cd8 f719a200b880a eb18fbcd2f21e49b871777d07ae46613ad701c6a78e6e72be0fcd10ff526abf4 9 hours ago Running prometheus 1 93f8761841c1e aa0b6df7616e5 f7d7982a2d4aced570c3c8c049fb4855e2d4b5d5508fb2e85c08e865a5a81675 9 hours ago Running prom-label-proxy 0 93f8761841c1e b84bc440a61da f62a4a4f08f8c9fb9b8f5580b6f8bdb6cb2ad17a8c33d42e35d173b39fe4365b 9 hours ago Running kube-rbac-proxy 0 93f8761841c1e 7452b4f0eabe6 bb681caf48d2958def5ee966bf01a49bcd9e9d7ba5d0e5abfeae94786ee4814e 9 hours ago Running thanos-sidecar 0 93f8761841c1e 9f147ad7b4b83 ee010c0ddb991e8fdb05bcc0f18c8289a24f8b534bf0b69796ffb6527ac42f40 9 hours ago Running rules-configmap-reloader 0 93f8761841c1e 7243d441df0ee 88334568560d52141e2840d314627d16cb8a2677eb5da7bafce964848fefaca1 9 hours ago Running prometheus-config-reloader 0 93f8761841c1e b71f560079341 3e2b86094e4ac64120d462e4d53e4b7c36254b662c875b0ab31850a52fb90256 16 hours ago Running metal3-static-ip-manager 0 0a78b1f410fa5 ca9352a7f5354 df051b2ab733377fca5ca322be7813189f6090b4790d31fcbca5555ebd87cee5 16 hours ago Running metal3-ironic-inspector 0 0a78b1f410fa5 c9d6106f9c63d c66d326237ff5bfe0286b7846caaa0e844b87a8f3fea4bb3225077f36954a194 16 hours ago Running metal3-ironic-api 0 0a78b1f410fa5 31570feb6a2a6 c66d326237ff5bfe0286b7846caaa0e844b87a8f3fea4bb3225077f36954a194 16 hours ago Running metal3-ironic-conductor 0 0a78b1f410fa5 f802ba1ea0365 c66d326237ff5bfe0286b7846caaa0e844b87a8f3fea4bb3225077f36954a194 16 hours ago Running metal3-httpd 0 0a78b1f410fa5 15a16896ef575 c66d326237ff5bfe0286b7846caaa0e844b87a8f3fea4bb3225077f36954a194 16 hours ago Running metal3-mariadb 0 0a78b1f410fa5 d6f6a056dbc69 c66d326237ff5bfe0286b7846caaa0e844b87a8f3fea4bb3225077f36954a194 16 hours ago Running metal3-dnsmasq 0 0a78b1f410fa5 99fe74469ef15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:309f8d90bbbc7571676a0ac7abead9370908df74fcc4dcc1705261691e9f56d7 16 hours ago Running prom-label-proxy 0 19a3f8fc696e6 ad706fe7c2410 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:309f8d90bbbc7571676a0ac7abead9370908df74fcc4dcc1705261691e9f56d7 16 hours ago Running prom-label-proxy 0 352afcc08d1da 2f15aa5898e63 f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 352afcc08d1da 27f8477cf046a f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 19a3f8fc696e6 d322b07b907e7 f62a4a4f08f8c9fb9b8f5580b6f8bdb6cb2ad17a8c33d42e35d173b39fe4365b 16 hours ago Running kube-rbac-proxy 0 10867ae65d781 d714eebb34abb ee010c0ddb991e8fdb05bcc0f18c8289a24f8b534bf0b69796ffb6527ac42f40 16 hours ago Running config-reloader 0 d6c765dbc4c78 de4fac9185b34 65b79ae4228a53a610e262033d535e24f8cd83cbb1d467fa3bd6224b2d66d0e5 16 hours ago Running openshift-state-metrics 0 b545bbddefb36 b572395f159a5 ee010c0ddb991e8fdb05bcc0f18c8289a24f8b534bf0b69796ffb6527ac42f40 16 hours ago Running reload 0 10867ae65d781 19fb7d3e7633a eada0a478bed63bc96723fa46ac32b0fa16642ee11a07e832aa6dda5afad895e 16 hours ago Running cluster-image-registry-operator-watch 0 44c5c2add508e f475b7534f508 f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 b072fa6748f5d 2831bb98d755a 3a3312cb7e6150b81d7e58c3514647a9a11e83ab95a7e2ff84998c7a64d78783 16 hours ago Running cluster-samples-operator-watch 0 19be5cb3ebb4b 61208c6d44936 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ddc3bdec7cd26e65f3e4139f37a6ff601da9551bdc4daf7fee7b4b7a5bf19cb0 16 hours ago Running redhat-operators 0 a171f8a67b3fa 8b99655eee068 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ddc3bdec7cd26e65f3e4139f37a6ff601da9551bdc4daf7fee7b4b7a5bf19cb0 16 hours ago Running community-operators 0 4a70752e4acb8 33bfcc3099415 f62a4a4f08f8c9fb9b8f5580b6f8bdb6cb2ad17a8c33d42e35d173b39fe4365b 16 hours ago Running kube-rbac-proxy-self 0 b545bbddefb36 f3df5221d58ec quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ddc3bdec7cd26e65f3e4139f37a6ff601da9551bdc4daf7fee7b4b7a5bf19cb0 16 hours ago Running certified-operators 0 1f6c53f0c745f b277ee5c10cdb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d0fcd6fe9aee3b995589e8f6f38179dd2310dc1c1fbf9bafad5c39ef94d990 16 hours ago Running grafana 0 88dfd45ebf426 96673f66fcfb7 29255c72af1e31b5b353815da5d18e554bb03ac811569e9320e865a60c5050db 16 hours ago Running alertmanager 0 d6c765dbc4c78 0ea4a89957b88 f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 cc57d67d2e689 6d2f033872cd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:279eb788fc275b7e5e98da14e6c1be82e8d1f5694981628dd4bca29f68ca79d5 16 hours ago Running network-operator 0 276c6031f514b 39ab61ae38962 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843a3c9fdea2eb96078e0030b67b8696df847b5b47b151ddeda67dd0155c37a9 16 hours ago Running cluster-image-registry-operator 0 44c5c2add508e 5aebff0190283 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2b519f7834eb10b1db5ec482275b59e445c57d1541786300bd704068de0d09a 16 hours ago Running machine-approver-controller 0 3424cedd25b99 c003f3c94c3c3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ddc3bdec7cd26e65f3e4139f37a6ff601da9551bdc4daf7fee7b4b7a5bf19cb0 16 hours ago Running redhat-marketplace 0 80c378ac8f65c ed564365e2aae quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19cfc3e37e12d9dd4e4dd9307781368bbeb07929b6ab788e99aa5543badee3c9 16 hours ago Running telemeter-client 0 10867ae65d781 7737f65c28187 f62a4a4f08f8c9fb9b8f5580b6f8bdb6cb2ad17a8c33d42e35d173b39fe4365b 16 hours ago Running kube-rbac-proxy-self 0 3e5af3fb262b7 2db066573fce7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e3548f71970f182836075330e34334c92bd2ffb1091520efea64f33778cc76a 16 hours ago Running migrator 0 7e639dc8cef68 cb80451cd004c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9870984bcbff1458d5293338f82caa512acc2b528bfab59c3be9056b42885fc 16 hours ago Running snapshot-controller 0 85269394a7aaa 2f96ad9d1ca24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20665686eba112407b8df6e7e7a2f739d5666b16e67fc4a468696734951b080 16 hours ago Running thanos-querier 0 352afcc08d1da 3b6a55a0f3842 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea32f7d002d2c5a761f68227417280c7fbf77db856f379af2d4bac57bd236d9 16 hours ago Running cluster-samples-operator 0 19be5cb3ebb4b 0eb0c773adaed quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20665686eba112407b8df6e7e7a2f739d5666b16e67fc4a468696734951b080 16 hours ago Running thanos-querier 0 19a3f8fc696e6 1594fa74b16fe f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 e3e85a45b8367 9865f410a9edf f62a4a4f08f8c9fb9b8f5580b6f8bdb6cb2ad17a8c33d42e35d173b39fe4365b 16 hours ago Running kube-rbac-proxy-main 0 b545bbddefb36 62bc39a65362b 481ca9e594d2f70f735494b2560082344eabad257630985b55be7091a71c7a74 16 hours ago Running download-server 0 7989705f4556a 6c0400d00c4c9 9c1b4201320164844e6af34ed8ca4654fc7fc6b5f320d3502faf1f9da4aafae2 16 hours ago Running machine-api-operator 0 12fb3a07949c4 0e76d4b919576 f62a4a4f08f8c9fb9b8f5580b6f8bdb6cb2ad17a8c33d42e35d173b39fe4365b 16 hours ago Running kube-rbac-proxy-main 0 3e5af3fb262b7 610e5d1109b92 f6bfa3b868f16a0231eff47755a5e7651d190355b03fa45397a9a87f6b20bcd1 16 hours ago Running machine-config-operator 0 82a26035cf1bc 09487a20daee3 f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 12fb3a07949c4 832c536358af2 f6bfa3b868f16a0231eff47755a5e7651d190355b03fa45397a9a87f6b20bcd1 16 hours ago Running machine-config-controller 0 76039e6015b21 5ec1517a0968d f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 3424cedd25b99 32f7e1b38252c 0a37b154df3dfcd0fa403af56a24c5e1beabf48c585d8a42cd63479425bf5b90 16 hours ago Running dns-node-resolver 0 16b8e431118d8 2fe3b8c4b5883 abaf0fb6ead1118bd61a6d2bf5f7fffc14f98dbd9f20f07341f85b6c6c7c4555 16 hours ago Running dns 0 16b8e431118d8 6d781eb7edf98 3b9ff8d043cbb4f087aec9d4d1f61540987bdfebbf0419931b0418853f108e38 16 hours ago Running machine-controller 0 70f8987ed01d0 e8254fc2e1525 f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 e20d3728c7188 99cac55b9e62f 5276a9fb191e97f6b88dc74a4bbd2958edcf92a36fb56a5c51b068476b093044 16 hours ago Running multus-admission-controller 0 e20d3728c7188 a904f3893c6f7 481ca9e594d2f70f735494b2560082344eabad257630985b55be7091a71c7a74 16 hours ago Running download-server 0 3dd960891be4f 9eed93b6d64a7 fab32138961aafcf4c48d1088ad65e6022dd9bd154027c67ed2b5fce8f54db52 16 hours ago Running controller-manager 0 9e6daf79c3e50 cfdd6ea7e45e4 3ef24d7faab2a25065550996a062644403a88817217b454941c0e09c386a5a78 16 hours ago Running kube-multus 0 7de1582ffcebd 48df1a6d52b6d f81493bcd63de826f08c722b67c0131242dee27ea90552fcd0c54b3feee10b16 16 hours ago Running kube-rbac-proxy 0 d8992dae5f832 c7150155cb7d9 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running ovnkube-master 0 558f7a9488ca4 3c37907805910 570876266458b84ea9983df71392f57c0c9012a24c3202de8a6ee782d8463b70 16 hours ago Running node-exporter 0 d8992dae5f832 f0858c199fbe8 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running sbdb 0 558f7a9488ca4 f8f02d3c2dd6a c8219e29d050a107f6985f7bd967279b1fd7ceb5fa83be3c142694d104f858bb 16 hours ago Running kube-controller-manager-recovery-controller 0 ccd97f003d5c0 fd9186cfc6f48 c8219e29d050a107f6985f7bd967279b1fd7ceb5fa83be3c142694d104f858bb 16 hours ago Running kube-controller-manager-cert-syncer 0 ccd97f003d5c0 55e56640a8c15 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running nbdb 0 558f7a9488ca4 fe3fa2391928a 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running ovnkube-node 0 99f65c13978a7 9fc2c7e3f29e4 0a37b154df3dfcd0fa403af56a24c5e1beabf48c585d8a42cd63479425bf5b90 16 hours ago Running guard 0 e0306cf847865 7b1c4ced2af22 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running northd 0 558f7a9488ca4 b077cb2dffcda f6bfa3b868f16a0231eff47755a5e7651d190355b03fa45397a9a87f6b20bcd1 16 hours ago Running machine-config-server 0 e56e3f6674c91 87d68e5001988 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running ovn-controller 0 99f65c13978a7 8e9e830c828fa 558504fd7771b0082f60671247ead83c2fd8b8f14491c1c99b99ec66dab40376 16 hours ago Running ovs-daemons 0 5df5524a6cfa8 af0918980b892 92085f2694455ad42f6e10ca2cea07d1b5e6c81d6e83cf7d16221a620ca55fb1 16 hours ago Running tuned 0 717b24b902f9a 6a8fea4312993 f6bfa3b868f16a0231eff47755a5e7651d190355b03fa45397a9a87f6b20bcd1 16 hours ago Running machine-config-daemon 0 ae9a2ede72629 81d562d80b066 ff92603cb90b741117c6192974d0a8a83e2b7abc1c49ba730ec72bad33130766 16 hours ago Running kube-controller-manager 0 ccd97f003d5c0 ec3a08b7793ff c4a16e5278a699ed0dbda97963ea20dfc89c621dff6799eda22b6ae94f7f8783 16 hours ago Running kube-scheduler-cert-syncer 0 4f0653544197e f78bbe1cc6931 ff92603cb90b741117c6192974d0a8a83e2b7abc1c49ba730ec72bad33130766 16 hours ago Running kube-scheduler 0 4f0653544197e 83b432572729c e607da46dc66b8c519585a65a6f6ce554affb576316123e8b01c0ab21da7d85e 16 hours ago Running kube-apiserver-insecure-readyz 0 8f712715ad402 b6a0738c001b9 e607da46dc66b8c519585a65a6f6ce554affb576316123e8b01c0ab21da7d85e 16 hours ago Running kube-apiserver-cert-regeneration-controller 0 8f712715ad402 a439f3ffcb1d6 e607da46dc66b8c519585a65a6f6ce554affb576316123e8b01c0ab21da7d85e 16 hours ago Running kube-apiserver-cert-syncer 0 8f712715ad402 074cbd2e32567 ff92603cb90b741117c6192974d0a8a83e2b7abc1c49ba730ec72bad33130766 16 hours ago Running kube-apiserver 0 8f712715ad402 96fcaab8e3748 70bb895d8d3e13543aebaf6fb9d6972fb76061a195c637a358b39853fb0e7fe8 16 hours ago Running etcd 0 4e2ea7b43b955 [core@master-1-0 ~]$ q
[core@master-1-0 ~]$ for i in `sudo crictl ps|awk '! /hours/ && ! /CONTAINER/ {print $1 $(NF-2)}'`; do echo "#################"; echo $i; done################# a374c5e5cbed1 openshift-apiserver ################# c24aa0abf691c operator ################# e006afbc18409 packageserver ################# 33548e6b2a24a kube-controller-manager-operator ################# 66222a460d40c console-operator ################# 261ef653fca43 cluster-node-tuning-operator ################# 9e57136d1b35c cluster-storage-operator ################# 0a9b587d5ed22 packageserver ################# 4e98976633cfc kube-scheduler-operator-container ################# 830ba7ae2761d operator ################# 0547ad6d0398c kube-apiserver-operator ################# be31c977539c9 cluster-policy-controller [core@master-1-0 ~]$ for i in `sudo crictl ps|awk '! /hours/ && ! /CONTAINER/ {print $1}'`; do echo "#################"; echo $i; sudo crictl logs $i 2>&1|grep -i -e fail -e error; done ################# a374c5e5cbed1 ################# c24aa0abf691c ################# e006afbc18409 ################# 33548e6b2a24a ################# 66222a460d40c ################# 261ef653fca43 ################# 9e57136d1b35c ################# 0a9b587d5ed22 ################# 4e98976633cfc ################# 830ba7ae2761d ################# 0547ad6d0398c ################# be31c977539c9 E0617 15:10:30.579846 1 reconciliation_controller.go:115] initial discovery check failure, continuing and counting on future sync update: unable to retrieve the complete list of server APIs: apps.openshift.io/v1: the server is currently unable to handle the request, authorization.openshift.io/v1: the server is currently unable to handle the request, build.openshift.io/v1: the server is currently unable to handle the request, image.openshift.io/v1: the server is currently unable to handle the request, metrics.k8s.io/v1beta1: the server is currently unable to handle the request, oauth.openshift.io/v1: the server is currently unable to handle the request, packages.operators.coreos.com/v1: the server is currently unable to handle the request, project.openshift.io/v1: the server is currently unable to handle the request, quota.openshift.io/v1: the server is currently unable to handle the request, route.openshift.io/v1: the server is currently unable to handle the request, security.openshift.io/v1: the server is currently unable to handle the request, template.openshift.io/v1: the server is currently unable to handle the request, user.openshift.io/v1: the server is currently unable to handle the request E0617 15:10:30.582377 1 reconciliation_controller.go:121] initial monitor sync has error: [couldn't start monitor for resource "monitoring.coreos.com/v1, Resource=podmonitors": unable to monitor quota for resource "monitoring.coreos.com/v1, Resource=podmonitors", couldn't start monitor for resource "multicloud.ibm.com/v1beta1, Resource=endpoints": unable to monitor quota for resource "multicloud.ibm.com/v1beta1, Resource=endpoints", couldn't start monitor for resource "ingress.operator.openshift.io/v1, Resource=dnsrecords": unable to monitor quota for resource "ingress.operator.openshift.io/v1, Resource=dnsrecords", couldn't start monitor for resource "monitoring.coreos.com/v1, Resource=prometheusrules": unable to monitor quota for resource "monitoring.coreos.com/v1, Resource=prometheusrules", couldn't start monitor for resource "tuned.openshift.io/v1, Resource=tuneds": unable to monitor quota for resource "tuned.openshift.io/v1, Resource=tuneds", couldn't start monitor for resource "autoscaling.openshift.io/v1beta1, Resource=machineautoscalers": unable to monitor quota for resource "autoscaling.openshift.io/v1beta1, Resource=machineautoscalers", couldn't start monitor for resource "network.operator.openshift.io/v1, Resource=operatorpkis": unable to monitor quota for resource "network.operator.openshift.io/v1, Resource=operatorpkis", couldn't start monitor for resource "operators.coreos.com/v1alpha1, Resource=catalogsources": unable to monitor quota for resource "operators.coreos.com/v1alpha1, Resource=catalogsources", couldn't start monitor for resource "cloudcredential.openshift.io/v1, Resource=credentialsrequests": unable to monitor quota for resource "cloudcredential.openshift.io/v1, Resource=credentialsrequests", couldn't start monitor for resource "operators.coreos.com/v1alpha1, Resource=subscriptions": unable to monitor quota for resource "operators.coreos.com/v1alpha1, Resource=subscriptions", couldn't start monitor for resource "machine.openshift.io/v1beta1, Resource=machinesets": unable to monitor quota for resource "machine.openshift.io/v1beta1, Resource=machinesets", couldn't start monitor for resource "monitoring.coreos.com/v1, Resource=prometheuses": unable to monitor quota for resource "monitoring.coreos.com/v1, Resource=prometheuses", couldn't start monitor for resource "machine.openshift.io/v1beta1, Resource=machines": unable to monitor quota for resource "machine.openshift.io/v1beta1, Resource=machines", couldn't start monitor for resource "machine.openshift.io/v1beta1, Resource=machinehealthchecks": unable to monitor quota for resource "machine.openshift.io/v1beta1, Resource=machinehealthchecks", couldn't start monitor for resource "k8s.cni.cncf.io/v1, Resource=network-attachment-definitions": unable to monitor quota for resource "k8s.cni.cncf.io/v1, Resource=network-attachment-definitions", couldn't start monitor for resource "metal3.io/v1alpha1, Resource=baremetalhosts": unable to monitor quota for resource "metal3.io/v1alpha1, Resource=baremetalhosts", couldn't start monitor for resource "operators.coreos.com/v1alpha1, Resource=clusterserviceversions": unable to monitor quota for resource "operators.coreos.com/v1alpha1, Resource=clusterserviceversions", couldn't start monitor for resource "tuned.openshift.io/v1, Resource=profiles": unable to monitor quota for resource "tuned.openshift.io/v1, Resource=profiles", couldn't start monitor for resource "operators.coreos.com/v2, Resource=catalogsourceconfigs": unable to monitor quota for resource "operators.coreos.com/v2, Resource=catalogsourceconfigs", couldn't start monitor for resource "operators.coreos.com/v1, Resource=operatorgroups": unable to monitor quota for resource "operators.coreos.com/v1, Resource=operatorgroups", couldn't start monitor for resource "whereabouts.cni.cncf.io/v1alpha1, Resource=ippools": unable to monitor quota for resource "whereabouts.cni.cncf.io/v1alpha1, Resource=ippools", couldn't start monitor for resource "operator.openshift.io/v1, Resource=ingresscontrollers": unable to monitor quota for resource "operator.openshift.io/v1, Resource=ingresscontrollers", couldn't start monitor for resource "monitoring.coreos.com/v1, Resource=alertmanagers": unable to monitor quota for resource "monitoring.coreos.com/v1, Resource=alertmanagers", couldn't start monitor for resource "snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshots": unable to monitor quota for resource "snapshot.storage.k8s.io/v1beta1, Resource=volumesnapshots", couldn't start monitor for resource "monitoring.coreos.com/v1, Resource=servicemonitors": unable to monitor quota for resource "monitoring.coreos.com/v1, Resource=servicemonitors", couldn't start monitor for resource "operators.coreos.com/v1, Resource=operatorsources": unable to monitor quota for resource "operators.coreos.com/v1, Resource=operatorsources", couldn't start monitor for resource "operators.coreos.com/v1alpha1, Resource=installplans": unable to monitor quota for resource "operators.coreos.com/v1alpha1, Resource=installplans"] E0617 15:10:30.696145 1 reflector.go:153] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: Failed to list *v1.ImageStream: the server is currently unable to handle the request (get imagestreams.image.openshift.io) E0617 15:10:31.698610 1 reflector.go:153] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: Failed to list *v1.ImageStream: the server is currently unable to handle the request (get imagestreams.image.openshift.io) E0617 15:10:32.701161 1 reflector.go:153] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: Failed to list *v1.ImageStream: the server is currently unable to handle the request (get imagestreams.image.openshift.io) E0617 15:10:33.703256 1 reflector.go:153] github.com/openshift/client-go/image/informers/externalversions/factory.go:101: Failed to list *v1.ImageStream: the server is currently unable to handle the request (get imagestreams.image.openshift.io)
I believe that you are hitting[1] which was resolved in 4.4.7. During restore we stop all static pods moving them to ${ASSET_DIR}/manifests-stopped. I believe if you move that static pod (networking) back into the manifests directory it should work. It should be located in [core@master-1-0 ~] ls ./assets/manifests-stopped [1] https://bugzilla.redhat.com/show_bug.cgi?id=1836270 Setting this to medium for now but if this is a production cluster with customer case we can escalate.
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.
Closing the bug as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1836270 *** This bug has been marked as a duplicate of bug 1836270 ***