Description of problem: This is on a baremetal install of 4.3.0 ======================================================================== [kni@f24-h06-000-r630 ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 95m Unable to apply 4.4.0: the cluster operator etcd is degraded ========================================================================= [kni@f24-h06-000-r630 ~]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.4.0 True False False 23m cloud-credential 4.4.0 True False False 92m cluster-autoscaler 4.4.0 True False False 57m console 4.4.0 True False False 26m csi-snapshot-controller 4.4.0 True False False 59m dns 4.4.0 True False False 63m etcd 4.4.0 False True True 65m image-registry 4.4.0 True False False 59m ingress 4.4.0 True False False 29m insights 4.4.0 True False False 59m kube-apiserver 4.4.0 True False False 61m kube-controller-manager 4.4.0 True False False 61m kube-scheduler 4.4.0 True False False 62m kube-storage-version-migrator 4.4.0 True False False 65m machine-api 4.4.0 True False False 64m machine-config 4.4.0 True False False 58m marketplace 4.4.0 True False False 58m monitoring 4.4.0 True False False 28m network 4.4.0 True False False 64m node-tuning 4.4.0 True False False 65m openshift-apiserver 4.4.0 True False False 59m openshift-controller-manager 4.4.0 True False False 58m openshift-samples 4.4.0 True False False 57m operator-lifecycle-manager 4.4.0 True False False 64m operator-lifecycle-manager-catalog 4.4.0 True False False 64m operator-lifecycle-manager-packageserver 4.4.0 True False False 59m service-ca 4.4.0 True False False 65m service-catalog-apiserver 4.4.0 True False False 65m service-catalog-controller-manager 4.4.0 True False False 65m storage 4.4.0 True False False 59m ======================================================================== [kni@f24-h06-000-r630 ~]$ oc project openshift-etcd Now using project "openshift-etcd" on server "https://api.test645.myocp4.com:6443". [kni@f24-h06-000-r630 ~]$ oc get pods NAME READY STATUS RESTARTS AGE etcd-master-0 3/3 Running 4 67m etcd-master-2 2/3 CrashLoopBackOff 17 65m installer-2-master-0 0/1 Completed 0 67m installer-3-master-2 0/1 Completed 0 65m revision-pruner-2-master-0 0/1 Completed 0 65m ===================================================================== [kni@f24-h06-000-r630 ~]$ oc project Using project "openshift-etcd" on server "https://api.test645.myocp4.com:6443". [kni@f24-h06-000-r630 ~]$ oc get pods NAME READY STATUS RESTARTS AGE etcd-master-0 3/3 Running 4 72m etcd-master-2 2/3 CrashLoopBackOff 18 69m installer-2-master-0 0/1 Completed 0 72m installer-3-master-2 0/1 Completed 0 70m revision-pruner-2-master-0 0/1 Completed 0 70m [kni@f24-h06-000-r630 ~]$ oc logs -c etcd etcd-master-2 52e70ae23e24c019, unstarted, , https://192.168.222.12:2380, b9b9806c92fe7a29, started, master-0, https://192.168.222.10:2380, https://192.168.222.10:2379 d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379 memberDir /var/lib/etcd/member is present on master-2 #### attempt 0 member={name="", peerURLs=[https://192.168.222.12:2380}, clientURLs=[] member={name="master-0", peerURLs=[https://192.168.222.10:2380}, clientURLs=[https://192.168.222.10:2379] member={name="etcd-bootstrap", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379] target={name="etcd-bootstrap", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379], err=<nil> mv: cannot stat '/etc/kubernetes/manifests/etcd-member.yaml': No such file or directory Waiting for ports 2379, 2380 and 9978 to be released.ETCD_PORT_9979_TCP=tcp://172.30.226.162:9979 ETCDCTL_CERT=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.crt ETCD_IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8f9978516adb30da807b5b30551348223827419ad0666905a6f8792bf51462c ETCD_PORT_9979_TCP_PORT=9979 ETCD_PORT_2379_TCP_PORT=2379 ETCD_PORT_2379_TCP_PROTO=tcp ETCD_INITIAL_CLUSTER_STATE=existing ETCD_PORT_9979_TCP_ADDR=172.30.226.162 ALL_ETCD_ENDPOINTS=https://192.168.222.10:2379,https://192.168.222.12:2379,https://192.168.222.11:2379,https://192.168.222.127:2379 ETCDCTL_ENDPOINTS=https://192.168.222.10:2379,https://192.168.222.12:2379,https://192.168.222.11:2379,https://192.168.222.127:2379 ETCD_INITIAL_CLUSTER=master-0=https://192.168.222.10:2380,etcd-bootstrap=https://192.168.222.127:2380 ETCD_ELECTION_TIMEOUT=1000 ETCD_SERVICE_PORT_ETCD=2379 ETCD_SERVICE_PORT_ETCD_METRICS=9979 ETCDCTL_CACERT=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt ETCD_NAME=master-2 ETCD_QUOTA_BACKEND_BYTES=7516192768 ETCD_SERVICE_PORT=2379 ETCD_PORT_2379_TCP_ADDR=172.30.226.162 ETCDCTL_API=3 ETCD_DATA_DIR=/var/lib/etcd ETCD_PORT_2379_TCP=tcp://172.30.226.162:2379 ETCD_PORT_9979_TCP_PROTO=tcp ETCD_PORT=tcp://172.30.226.162:2379 ETCD_HEARTBEAT_INTERVAL=100 ETCDCTL_KEY=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.key ETCD_SERVICE_HOST=172.30.226.162 + exec etcd --initial-advertise-peer-urls=https://192.168.222.12:2380 --cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-serving/etcd-serving-master-2.crt --key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-serving/etcd-serving-master-2.key --trusted-ca-file=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt --client-cert-auth=true --peer-cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.crt --peer-key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.key --peer-trusted-ca-file=/etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt --peer-client-cert-auth=true --advertise-client-urls=https://192.168.222.12:2379 --listen-client-urls=https://0.0.0.0:2379 --listen-peer-urls=https://0.0.0.0:2380 --listen-metrics-urls=https://0.0.0.0:9978 2020-05-07 17:06:34.678148 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd 2020-05-07 17:06:34.678203 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1000 2020-05-07 17:06:34.678230 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=100 2020-05-07 17:06:34.678240 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=master-0=https://192.168.222.10:2380,etcd-bootstrap=https://192.168.222.127:2380 2020-05-07 17:06:34.678245 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing 2020-05-07 17:06:34.678259 I | pkg/flags: recognized and used environment variable ETCD_NAME=master-2 2020-05-07 17:06:34.678273 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=7516192768 2020-05-07 17:06:34.678291 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP=tcp://172.30.226.162:9979 2020-05-07 17:06:34.678296 W | pkg/flags: unrecognized environment variable ETCD_IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8f9978516adb30da807b5b30551348223827419ad0666905a6f8792bf51462c 2020-05-07 17:06:34.678300 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP_PORT=9979 2020-05-07 17:06:34.678303 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379 2020-05-07 17:06:34.678306 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp 2020-05-07 17:06:34.678310 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP_ADDR=172.30.226.162 2020-05-07 17:06:34.678315 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_ETCD=2379 2020-05-07 17:06:34.678320 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_ETCD_METRICS=9979 2020-05-07 17:06:34.678326 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379 2020-05-07 17:06:34.678329 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=172.30.226.162 2020-05-07 17:06:34.678333 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://172.30.226.162:2379 2020-05-07 17:06:34.678337 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP_PROTO=tcp 2020-05-07 17:06:34.678340 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://172.30.226.162:2379 2020-05-07 17:06:34.678344 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=172.30.226.162 2020-05-07 17:06:34.678366 I | etcdmain: etcd Version: 3.3.18 2020-05-07 17:06:34.678373 I | etcdmain: Git SHA: c0157a9 2020-05-07 17:06:34.678376 I | etcdmain: Go Version: go1.13.4 2020-05-07 17:06:34.678379 I | etcdmain: Go OS/Arch: linux/amd64 2020-05-07 17:06:34.678383 I | etcdmain: setting maximum number of CPUs to 56, total number of available CPUs is 56 2020-05-07 17:06:34.678425 N | etcdmain: the server is already initialized as member before, starting as etcd member... 2020-05-07 17:06:34.678450 I | embed: peerTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.key, ca = , trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt, client-cert-auth = true, crl-file = 2020-05-07 17:06:34.678958 I | embed: listening for peers on https://0.0.0.0:2380 2020-05-07 17:06:34.679025 I | embed: listening for client requests on 0.0.0.0:2379 2020-05-07 17:06:34.682789 I | embed: rejected connection from "192.168.222.127:43020" (error "set tcp 192.168.222.12:2380: use of closed network connection", ServerName "") 2020-05-07 17:06:34.682828 C | etcdmain: couldn't find local name "master-2" in the initial cluster configuration ============================================================================ [kni@f24-h06-000-r630 ~]$ oc logs etcd-master-2 -c etcd-ensure-env-vars [kni@f24-h06-000-r630 ~]$ ============================================================================= Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: [kni@f24-h06-000-r630 ~]$ oc project openshift-etcd-operator Already on project "openshift-etcd-operator" on server "https://api.test645.myocp4.com:6443". [kni@f24-h06-000-r630 ~]$ oc get events LAST SEEN TYPE REASON OBJECT MESSAGE <unknown> Warning FailedScheduling pod/etcd-operator-58f4bd7c4d-glzv6 no nodes available to schedule pods <unknown> Warning FailedScheduling pod/etcd-operator-58f4bd7c4d-glzv6 no nodes available to schedule pods <unknown> Warning FailedScheduling pod/etcd-operator-58f4bd7c4d-glzv6 no nodes available to schedule pods <unknown> Warning FailedScheduling pod/etcd-operator-58f4bd7c4d-glzv6 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. <unknown> Warning FailedScheduling pod/etcd-operator-58f4bd7c4d-glzv6 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate. <unknown> Warning FailedScheduling pod/etcd-operator-58f4bd7c4d-glzv6 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate. <unknown> Normal Scheduled pod/etcd-operator-58f4bd7c4d-glzv6 Successfully assigned openshift-etcd-operator/etcd-operator-58f4bd7c4d-glzv6 to master-1 74m Normal Pulling pod/etcd-operator-58f4bd7c4d-glzv6 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8fa5c557cc88d76148e963f914258323f7d3ec23ca9fb59c8516a50bbb90e858" 73m Normal Pulled pod/etcd-operator-58f4bd7c4d-glzv6 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8fa5c557cc88d76148e963f914258323f7d3ec23ca9fb59c8516a50bbb90e858" 72m Normal Created pod/etcd-operator-58f4bd7c4d-glzv6 Created container operator 72m Normal Started pod/etcd-operator-58f4bd7c4d-glzv6 Started container operator 72m Normal Pulled pod/etcd-operator-58f4bd7c4d-glzv6 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8fa5c557cc88d76148e963f914258323f7d3ec23ca9fb59c8516a50bbb90e858" already present on machine 101m Normal SuccessfulCreate replicaset/etcd-operator-58f4bd7c4d Created pod: etcd-operator-58f4bd7c4d-glzv6 101m Normal ScalingReplicaSet deployment/etcd-operator Scaled up replica set etcd-operator-58f4bd7c4d to 1 73m Warning ReportEtcdMembersErrorUpdatingStatus deployment/etcd-operator etcds.operator.openshift.io "cluster" not found 73m Normal OperatorLoglevelChange deployment/etcd-operator Operator log level changed from "Debug" to "Normal" 72m Warning MissingVersion deployment/etcd-operator no image found for operand pod 73m Warning RequiredInstallerResourcesMissing deployment/etcd-operator configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0 73m Normal OperatorVersionChanged deployment/etcd-operator clusteroperator/etcd version "raw-internal" changed from "" to "4.4.0" 73m Normal OperatorVersionChanged deployment/etcd-operator clusteroperator/etcd version "operator" changed from "" to "4.4.0" 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded set to False ("EtcdMembersControllerDegraded: node lister not synced"),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to True (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "etcds" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-etcd-operator"} {"" "namespaces" "" "openshift-etcd"}] 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Upgradeable changed from Unknown to True ("") 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced" 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values" 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]",Progressing changed from Unknown to False (""),Available changed from Unknown to False ("StaticPodsAvailable: 0 nodes are active; ") 73m Normal MasterNodeObserved deployment/etcd-operator Observed new master node master-0 73m Normal MasterNodeObserved deployment/etcd-operator Observed new master node master-2 73m Normal MasterNodeObserved deployment/etcd-operator Observed new master node master-1 73m Warning EndpointsUpdated deployment/etcd-operator Updated endpoints/host-etcd-2 -n openshift-etcd because it changed: {"subsets":[{"addresses":[{"ip":"192.168.222.10","nodeName":"master-0"},{"ip":"192.168.222.12","nodeName":"master-2"},{"ip":"192.168.222.11","nodeName":"master-1"}],"ports":[{"name":"etcd","port":2379,"protocol":"TCP"}]}]} 73m Normal RevisionTriggered deployment/etcd-operator new revision 1 triggered by "configmap \"etcd-pod\" not found" 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-peer-master-0 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-serving-master-0 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-serving-metrics-master-0 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-peer-master-2 -n openshift-etcd because it was missing 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-serving-master-2 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-serving-metrics-master-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/revision-status-1 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-peer-master-1 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-metrics-proxy-client-ca -n openshift-etcd because it was missing 73m Normal ServiceAccountCreated deployment/etcd-operator Created ServiceAccount/installer-sa -n openshift-etcd because it was missing 73m Normal ClusterRoleBindingCreated deployment/etcd-operator Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:openshift-etcd-installer because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-serving-master-1 -n openshift-etcd because it was missing 73m Warning RequiredInstallerResourcesMissing deployment/etcd-operator configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Progressing message changed from "" to "NodeInstallerProgressing: 3 nodes are at revision 0",Available message changed from "StaticPodsAvailable: 0 nodes are active; " to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0" 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-serving-metrics-master-1 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-serving-ca -n openshift-etcd because it was missing 73m Normal ServiceAccountCreated deployment/etcd-operator Created ServiceAccount/etcd-sa -n openshift-etcd because it was missing 73m Warning RequiredInstallerResourcesMissing deployment/etcd-operator configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-metrics-proxy-serving-ca -n openshift-etcd because it was missing 73m Warning EnvVarControllerUpdatingStatus deployment/etcd-operator Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again 73m Warning HostEndpoints2ErrorUpdatingStatus deployment/etcd-operator Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again 73m Normal ConfigMapUpdated deployment/etcd-operator Updated ConfigMap/etcd-ca-bundle -n openshift-etcd-operator: cause by changes in data.ca-bundle.crt 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-ca-bundle -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-scripts -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/config -n openshift-etcd because it was missing 73m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found" 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-peer-client-ca -n openshift-etcd because it was missing 73m Normal SecretUpdated deployment/etcd-operator Updated Secret/etcd-client -n openshift-etcd-operator because it changed 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-pod -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-client -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/restore-etcd-pod -n openshift-etcd because it was missing 73m Normal RevisionTriggered deployment/etcd-operator new revision 2 triggered by "configmap \"etcd-pod-1\" not found" 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/revision-status-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-pod-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/config-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-serving-ca-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-peer-client-ca-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-metrics-proxy-serving-ca-2 -n openshift-etcd because it was missing 73m Normal ConfigMapCreated deployment/etcd-operator Created ConfigMap/etcd-metrics-proxy-client-ca-2 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-all-peer-2 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-all-serving-2 -n openshift-etcd because it was missing 73m Normal SecretCreated deployment/etcd-operator Created Secret/etcd-all-serving-metrics-2 -n openshift-etcd because it was missing 73m Normal OperatorStatusChanged deployment/etcd-operator (combined from similar events): Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pods \"etcd-master-0\" not found\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pods \"etcd-master-0\" not found\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: All master nodes are ready" 73m Warning RequiredInstallerResourcesMissing deployment/etcd-operator configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1 73m Normal SecretCreated deployment/etcd-operator (combined from similar events): Created Secret/etcd-all-serving-metrics -n openshift-etcd because it was missing 73m Warning ScriptControllerErrorUpdatingStatus deployment/etcd-operator Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again 73m Warning BootstrapTeardownErrorUpdatingStatus deployment/etcd-operator Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again 73m Normal RevisionCreate deployment/etcd-operator Revision 1 created because configmap "etcd-pod-1" not found 73m Normal NodeTargetRevisionChanged deployment/etcd-operator Updating node "master-0" from revision 0 to 2 because node master-0 static pod not found 73m Warning ClusterMemberControllerUpdatingStatus deployment/etcd-operator Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again 72m Normal PodCreated deployment/etcd-operator Created Pod/installer-2-master-0 -n openshift-etcd because it was missing 71m Warning ReportEtcdMembersErrorUpdatingStatus deployment/etcd-operator etcds.operator.openshift.io "cluster" not found 71m Normal RevisionTriggered deployment/etcd-operator new revision 3 triggered by "configmap \"etcd-pod\" not found" 71m Normal OperatorLoglevelChange deployment/etcd-operator Operator log level changed from "Debug" to "Normal" 71m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded changed from False to True ("EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is not ready\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is terminated: \"Error\" - \"d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379\\n#### attempt 0\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 1\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 2\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 3\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 4\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 5\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 6\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 7\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 8\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 9\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\ntimed out\"\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nEtcdMemberIPMigratorDegraded: context deadline exceeded") 71m Normal OperatorStatusChanged deployment/etcd-operator Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is not ready\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is terminated: \"Error\" - \"d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379\\n#### attempt 0\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 1\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 2\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 3\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 4\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 5\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 6\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 7\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 8\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 9\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\ntimed out\"\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nEtcdMemberIPMigratorDegraded: context deadline exceeded" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is not ready\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is terminated: \"Error\" - \"d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379\\n#### attempt 0\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 1\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 2\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 3\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 4\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 5\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 6\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 7\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 8\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 9\\n member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n target=nil, err=<nil>\\n#### sleeping...\\ntimed out\"\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nEtcdMemberIPMigratorDegraded: context deadline exceeded"
Created attachment 1686261 [details] etcd-operator.log
This problem looks very similar to another we are chasing on bare metal. Closing it as duplicate to 1814576. *** This bug has been marked as a duplicate of bug 1814576 ***