Bug 1833050

Summary:

discover-etcd-initial-cluster results in invalid initial cluster configuration

Product:

OpenShift Container Platform

Reporter:

Sai Sindhur Malleni <smalleni>

Component:

Etcd Operator

Assignee:

Dan Mace <dmace>

Status:

CLOSED DUPLICATE

QA Contact:

ge liu <geliu>

Severity:

high

Docs Contact:

Priority:

high

Version:

4.4

CC:

skolicha

Target Milestone:

---

Target Release:

4.5.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-05-20 13:00:56 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
etcd-operator.log	none

Description Sai Sindhur Malleni 2020-05-07 17:39:41 UTC

Description of problem:

This is on a baremetal install of 4.3.0
========================================================================
[kni@f24-h06-000-r630 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          95m     Unable to apply 4.4.0: the cluster operator etcd is degraded

=========================================================================

[kni@f24-h06-000-r630 ~]$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.0     True        False         False      23m
cloud-credential                           4.4.0     True        False         False      92m
cluster-autoscaler                         4.4.0     True        False         False      57m
console                                    4.4.0     True        False         False      26m
csi-snapshot-controller                    4.4.0     True        False         False      59m
dns                                        4.4.0     True        False         False      63m
etcd                                       4.4.0     False       True          True       65m
image-registry                             4.4.0     True        False         False      59m
ingress                                    4.4.0     True        False         False      29m
insights                                   4.4.0     True        False         False      59m
kube-apiserver                             4.4.0     True        False         False      61m
kube-controller-manager                    4.4.0     True        False         False      61m
kube-scheduler                             4.4.0     True        False         False      62m
kube-storage-version-migrator              4.4.0     True        False         False      65m
machine-api                                4.4.0     True        False         False      64m
machine-config                             4.4.0     True        False         False      58m
marketplace                                4.4.0     True        False         False      58m
monitoring                                 4.4.0     True        False         False      28m
network                                    4.4.0     True        False         False      64m
node-tuning                                4.4.0     True        False         False      65m
openshift-apiserver                        4.4.0     True        False         False      59m
openshift-controller-manager               4.4.0     True        False         False      58m
openshift-samples                          4.4.0     True        False         False      57m
operator-lifecycle-manager                 4.4.0     True        False         False      64m
operator-lifecycle-manager-catalog         4.4.0     True        False         False      64m
operator-lifecycle-manager-packageserver   4.4.0     True        False         False      59m
service-ca                                 4.4.0     True        False         False      65m
service-catalog-apiserver                  4.4.0     True        False         False      65m
service-catalog-controller-manager         4.4.0     True        False         False      65m
storage                                    4.4.0     True        False         False      59m
========================================================================
[kni@f24-h06-000-r630 ~]$ oc project openshift-etcd
Now using project "openshift-etcd" on server "https://api.test645.myocp4.com:6443".
[kni@f24-h06-000-r630 ~]$ oc get pods
NAME                         READY   STATUS             RESTARTS   AGE
etcd-master-0                3/3     Running            4          67m
etcd-master-2                2/3     CrashLoopBackOff   17         65m
installer-2-master-0         0/1     Completed          0          67m
installer-3-master-2         0/1     Completed          0          65m
revision-pruner-2-master-0   0/1     Completed          0          65m

=====================================================================
[kni@f24-h06-000-r630 ~]$ oc project
Using project "openshift-etcd" on server "https://api.test645.myocp4.com:6443".
[kni@f24-h06-000-r630 ~]$ oc get pods
NAME                         READY   STATUS             RESTARTS   AGE
etcd-master-0                3/3     Running            4          72m
etcd-master-2                2/3     CrashLoopBackOff   18         69m
installer-2-master-0         0/1     Completed          0          72m
installer-3-master-2         0/1     Completed          0          70m
revision-pruner-2-master-0   0/1     Completed          0          70m
[kni@f24-h06-000-r630 ~]$ oc logs -c etcd etcd-master-2
52e70ae23e24c019, unstarted, , https://192.168.222.12:2380, 
b9b9806c92fe7a29, started, master-0, https://192.168.222.10:2380, https://192.168.222.10:2379
d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379
memberDir /var/lib/etcd/member is present on master-2
#### attempt 0
      member={name="", peerURLs=[https://192.168.222.12:2380}, clientURLs=[]
      member={name="master-0", peerURLs=[https://192.168.222.10:2380}, clientURLs=[https://192.168.222.10:2379]
      member={name="etcd-bootstrap", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]
      target={name="etcd-bootstrap", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379], err=<nil>
mv: cannot stat '/etc/kubernetes/manifests/etcd-member.yaml': No such file or directory
Waiting for ports 2379, 2380 and 9978 to be released.ETCD_PORT_9979_TCP=tcp://172.30.226.162:9979
ETCDCTL_CERT=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.crt
ETCD_IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8f9978516adb30da807b5b30551348223827419ad0666905a6f8792bf51462c
ETCD_PORT_9979_TCP_PORT=9979
ETCD_PORT_2379_TCP_PORT=2379
ETCD_PORT_2379_TCP_PROTO=tcp
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_PORT_9979_TCP_ADDR=172.30.226.162
ALL_ETCD_ENDPOINTS=https://192.168.222.10:2379,https://192.168.222.12:2379,https://192.168.222.11:2379,https://192.168.222.127:2379
ETCDCTL_ENDPOINTS=https://192.168.222.10:2379,https://192.168.222.12:2379,https://192.168.222.11:2379,https://192.168.222.127:2379
ETCD_INITIAL_CLUSTER=master-0=https://192.168.222.10:2380,etcd-bootstrap=https://192.168.222.127:2380
ETCD_ELECTION_TIMEOUT=1000
ETCD_SERVICE_PORT_ETCD=2379
ETCD_SERVICE_PORT_ETCD_METRICS=9979
ETCDCTL_CACERT=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt
ETCD_NAME=master-2
ETCD_QUOTA_BACKEND_BYTES=7516192768
ETCD_SERVICE_PORT=2379
ETCD_PORT_2379_TCP_ADDR=172.30.226.162
ETCDCTL_API=3
ETCD_DATA_DIR=/var/lib/etcd
ETCD_PORT_2379_TCP=tcp://172.30.226.162:2379
ETCD_PORT_9979_TCP_PROTO=tcp
ETCD_PORT=tcp://172.30.226.162:2379
ETCD_HEARTBEAT_INTERVAL=100
ETCDCTL_KEY=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.key
ETCD_SERVICE_HOST=172.30.226.162
+ exec etcd --initial-advertise-peer-urls=https://192.168.222.12:2380 --cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-serving/etcd-serving-master-2.crt --key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-serving/etcd-serving-master-2.key --trusted-ca-file=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt --client-cert-auth=true --peer-cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.crt --peer-key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.key --peer-trusted-ca-file=/etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt --peer-client-cert-auth=true --advertise-client-urls=https://192.168.222.12:2379 --listen-client-urls=https://0.0.0.0:2379 --listen-peer-urls=https://0.0.0.0:2380 --listen-metrics-urls=https://0.0.0.0:9978
2020-05-07 17:06:34.678148 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
2020-05-07 17:06:34.678203 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1000
2020-05-07 17:06:34.678230 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=100
2020-05-07 17:06:34.678240 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=master-0=https://192.168.222.10:2380,etcd-bootstrap=https://192.168.222.127:2380
2020-05-07 17:06:34.678245 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
2020-05-07 17:06:34.678259 I | pkg/flags: recognized and used environment variable ETCD_NAME=master-2
2020-05-07 17:06:34.678273 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=7516192768
2020-05-07 17:06:34.678291 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP=tcp://172.30.226.162:9979
2020-05-07 17:06:34.678296 W | pkg/flags: unrecognized environment variable ETCD_IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8f9978516adb30da807b5b30551348223827419ad0666905a6f8792bf51462c
2020-05-07 17:06:34.678300 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP_PORT=9979
2020-05-07 17:06:34.678303 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
2020-05-07 17:06:34.678306 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
2020-05-07 17:06:34.678310 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP_ADDR=172.30.226.162
2020-05-07 17:06:34.678315 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_ETCD=2379
2020-05-07 17:06:34.678320 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_ETCD_METRICS=9979
2020-05-07 17:06:34.678326 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
2020-05-07 17:06:34.678329 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=172.30.226.162
2020-05-07 17:06:34.678333 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://172.30.226.162:2379
2020-05-07 17:06:34.678337 W | pkg/flags: unrecognized environment variable ETCD_PORT_9979_TCP_PROTO=tcp
2020-05-07 17:06:34.678340 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://172.30.226.162:2379
2020-05-07 17:06:34.678344 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=172.30.226.162
2020-05-07 17:06:34.678366 I | etcdmain: etcd Version: 3.3.18
2020-05-07 17:06:34.678373 I | etcdmain: Git SHA: c0157a9
2020-05-07 17:06:34.678376 I | etcdmain: Go Version: go1.13.4
2020-05-07 17:06:34.678379 I | etcdmain: Go OS/Arch: linux/amd64
2020-05-07 17:06:34.678383 I | etcdmain: setting maximum number of CPUs to 56, total number of available CPUs is 56
2020-05-07 17:06:34.678425 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2020-05-07 17:06:34.678450 I | embed: peerTLS: cert = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.crt, key = /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master-2.key, ca = , trusted-ca = /etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt, client-cert-auth = true, crl-file = 
2020-05-07 17:06:34.678958 I | embed: listening for peers on https://0.0.0.0:2380
2020-05-07 17:06:34.679025 I | embed: listening for client requests on 0.0.0.0:2379
2020-05-07 17:06:34.682789 I | embed: rejected connection from "192.168.222.127:43020" (error "set tcp 192.168.222.12:2380: use of closed network connection", ServerName "")
2020-05-07 17:06:34.682828 C | etcdmain: couldn't find local name "master-2" in the initial cluster configuration
============================================================================
[kni@f24-h06-000-r630 ~]$ oc logs etcd-master-2 -c etcd-ensure-env-vars
[kni@f24-h06-000-r630 ~]$ 

=============================================================================

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
[kni@f24-h06-000-r630 ~]$ oc project openshift-etcd-operator
Already on project "openshift-etcd-operator" on server "https://api.test645.myocp4.com:6443".
[kni@f24-h06-000-r630 ~]$ oc get events
LAST SEEN   TYPE      REASON                                 OBJECT                                MESSAGE
<unknown>   Warning   FailedScheduling                       pod/etcd-operator-58f4bd7c4d-glzv6    no nodes available to schedule pods
<unknown>   Warning   FailedScheduling                       pod/etcd-operator-58f4bd7c4d-glzv6    no nodes available to schedule pods
<unknown>   Warning   FailedScheduling                       pod/etcd-operator-58f4bd7c4d-glzv6    no nodes available to schedule pods
<unknown>   Warning   FailedScheduling                       pod/etcd-operator-58f4bd7c4d-glzv6    0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
<unknown>   Warning   FailedScheduling                       pod/etcd-operator-58f4bd7c4d-glzv6    0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
<unknown>   Warning   FailedScheduling                       pod/etcd-operator-58f4bd7c4d-glzv6    0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
<unknown>   Normal    Scheduled                              pod/etcd-operator-58f4bd7c4d-glzv6    Successfully assigned openshift-etcd-operator/etcd-operator-58f4bd7c4d-glzv6 to master-1
74m         Normal    Pulling                                pod/etcd-operator-58f4bd7c4d-glzv6    Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8fa5c557cc88d76148e963f914258323f7d3ec23ca9fb59c8516a50bbb90e858"
73m         Normal    Pulled                                 pod/etcd-operator-58f4bd7c4d-glzv6    Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8fa5c557cc88d76148e963f914258323f7d3ec23ca9fb59c8516a50bbb90e858"
72m         Normal    Created                                pod/etcd-operator-58f4bd7c4d-glzv6    Created container operator
72m         Normal    Started                                pod/etcd-operator-58f4bd7c4d-glzv6    Started container operator
72m         Normal    Pulled                                 pod/etcd-operator-58f4bd7c4d-glzv6    Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8fa5c557cc88d76148e963f914258323f7d3ec23ca9fb59c8516a50bbb90e858" already present on machine
101m        Normal    SuccessfulCreate                       replicaset/etcd-operator-58f4bd7c4d   Created pod: etcd-operator-58f4bd7c4d-glzv6
101m        Normal    ScalingReplicaSet                      deployment/etcd-operator              Scaled up replica set etcd-operator-58f4bd7c4d to 1
73m         Warning   ReportEtcdMembersErrorUpdatingStatus   deployment/etcd-operator              etcds.operator.openshift.io "cluster" not found
73m         Normal    OperatorLoglevelChange                 deployment/etcd-operator              Operator log level changed from "Debug" to "Normal"
72m         Warning   MissingVersion                         deployment/etcd-operator              no image found for operand pod
73m         Warning   RequiredInstallerResourcesMissing      deployment/etcd-operator              configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0
73m         Normal    OperatorVersionChanged                 deployment/etcd-operator              clusteroperator/etcd version "raw-internal" changed from "" to "4.4.0"
73m         Normal    OperatorVersionChanged                 deployment/etcd-operator              clusteroperator/etcd version "operator" changed from "" to "4.4.0"
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded set to False ("EtcdMembersControllerDegraded: node lister not synced"),Progressing set to Unknown (""),Available set to Unknown (""),Upgradeable set to True (""),status.relatedObjects changed from [] to [{"operator.openshift.io" "etcds" "" "cluster"} {"" "namespaces" "" "openshift-config"} {"" "namespaces" "" "openshift-config-managed"} {"" "namespaces" "" "openshift-etcd-operator"} {"" "namespaces" "" "openshift-etcd"}]
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Upgradeable changed from Unknown to True ("")
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced"
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values"
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]",Progressing changed from Unknown to False (""),Available changed from Unknown to False ("StaticPodsAvailable: 0 nodes are active; ")
73m         Normal    MasterNodeObserved                     deployment/etcd-operator              Observed new master node master-0
73m         Normal    MasterNodeObserved                     deployment/etcd-operator              Observed new master node master-2
73m         Normal    MasterNodeObserved                     deployment/etcd-operator              Observed new master node master-1
73m         Warning   EndpointsUpdated                       deployment/etcd-operator              Updated endpoints/host-etcd-2 -n openshift-etcd because it changed: {"subsets":[{"addresses":[{"ip":"192.168.222.10","nodeName":"master-0"},{"ip":"192.168.222.12","nodeName":"master-2"},{"ip":"192.168.222.11","nodeName":"master-1"}],"ports":[{"name":"etcd","port":2379,"protocol":"TCP"}]}]}
73m         Normal    RevisionTriggered                      deployment/etcd-operator              new revision 1 triggered by "configmap \"etcd-pod\" not found"
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-peer-master-0 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-serving-master-0 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-serving-metrics-master-0 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-peer-master-2 -n openshift-etcd because it was missing
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)"
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-serving-master-2 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-serving-metrics-master-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                       deployment/etcd-operator              Created ConfigMap/revision-status-1 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-peer-master-1 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                       deployment/etcd-operator              Created ConfigMap/etcd-metrics-proxy-client-ca -n openshift-etcd because it was missing
73m         Normal    ServiceAccountCreated                  deployment/etcd-operator              Created ServiceAccount/installer-sa -n openshift-etcd because it was missing
73m         Normal    ClusterRoleBindingCreated              deployment/etcd-operator              Created ClusterRoleBinding.rbac.authorization.k8s.io/system:openshift:operator:openshift-etcd-installer because it was missing
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-serving-master-1 -n openshift-etcd because it was missing
73m         Warning   RequiredInstallerResourcesMissing      deployment/etcd-operator              configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Progressing message changed from "" to "NodeInstallerProgressing: 3 nodes are at revision 0",Available message changed from "StaticPodsAvailable: 0 nodes are active; " to "StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0"
73m         Normal    SecretCreated                          deployment/etcd-operator              Created Secret/etcd-serving-metrics-master-1 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                       deployment/etcd-operator              Created ConfigMap/etcd-serving-ca -n openshift-etcd because it was missing
73m         Normal    ServiceAccountCreated                  deployment/etcd-operator              Created ServiceAccount/etcd-sa -n openshift-etcd because it was missing
73m         Warning   RequiredInstallerResourcesMissing      deployment/etcd-operator              configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0
73m         Normal    OperatorStatusChanged                  deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-client-ca,etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)"
73m         Normal    ConfigMapCreated                       deployment/etcd-operator              Created ConfigMap/etcd-metrics-proxy-serving-ca -n openshift-etcd because it was missing
73m         Warning   EnvVarControllerUpdatingStatus         deployment/etcd-operator              Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
73m         Warning   HostEndpoints2ErrorUpdatingStatus      deployment/etcd-operator              Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
73m         Normal    ConfigMapUpdated                       deployment/etcd-operator              Updated ConfigMap/etcd-ca-bundle -n openshift-etcd-operator:
cause by changes in data.ca-bundle.crt
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-ca-bundle -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-scripts -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/config -n openshift-etcd because it was missing
73m         Normal    OperatorStatusChanged                   deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,etcd-serving-ca,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)\nRevisionControllerDegraded: configmaps \"etcd-pod\" not found"
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-peer-client-ca -n openshift-etcd because it was missing
73m         Normal    SecretUpdated                           deployment/etcd-operator              Updated Secret/etcd-client -n openshift-etcd-operator because it changed
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-pod -n openshift-etcd because it was missing
73m         Normal    SecretCreated                           deployment/etcd-operator              Created Secret/etcd-client -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/restore-etcd-pod -n openshift-etcd because it was missing
73m         Normal    RevisionTriggered                       deployment/etcd-operator              new revision 2 triggered by "configmap \"etcd-pod-1\" not found"
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/revision-status-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-pod-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/config-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-serving-ca-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-peer-client-ca-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-metrics-proxy-serving-ca-2 -n openshift-etcd because it was missing
73m         Normal    ConfigMapCreated                        deployment/etcd-operator              Created ConfigMap/etcd-metrics-proxy-client-ca-2 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                           deployment/etcd-operator              Created Secret/etcd-all-peer-2 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                           deployment/etcd-operator              Created Secret/etcd-all-serving-2 -n openshift-etcd because it was missing
73m         Normal    SecretCreated                           deployment/etcd-operator              Created Secret/etcd-all-serving-metrics-2 -n openshift-etcd because it was missing
73m         Normal    OperatorStatusChanged                   deployment/etcd-operator              (combined from similar events): Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pods \"etcd-master-0\" not found\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: The master nodes not ready: node \"master-0\" not ready since 2020-05-07 15:50:19 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?), node \"master-2\" not ready since 2020-05-07 15:50:24 +0000 UTC because KubeletNotReady (runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: pods \"etcd-master-0\" not found\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nInstallerControllerDegraded: missing required resources: [configmaps: etcd-metrics-proxy-serving-ca,etcd-peer-client-ca,etcd-scripts,restore-etcd-pod, configmaps: config-0,etcd-metrics-proxy-client-ca-0,etcd-metrics-proxy-serving-ca-0,etcd-peer-client-ca-0,etcd-pod-0,etcd-serving-ca-0, secrets: etcd-all-peer,etcd-all-serving,etcd-all-serving-metrics, secrets: etcd-all-peer-0,etcd-all-serving-0,etcd-all-serving-metrics-0]\nNodeControllerDegraded: All master nodes are ready"
73m         Warning   RequiredInstallerResourcesMissing       deployment/etcd-operator              configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1
73m         Normal    SecretCreated                           deployment/etcd-operator              (combined from similar events): Created Secret/etcd-all-serving-metrics -n openshift-etcd because it was missing
73m         Warning   ScriptControllerErrorUpdatingStatus     deployment/etcd-operator              Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
73m         Warning   BootstrapTeardownErrorUpdatingStatus    deployment/etcd-operator              Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
73m         Normal    RevisionCreate                          deployment/etcd-operator              Revision 1 created because configmap "etcd-pod-1" not found
73m         Normal    NodeTargetRevisionChanged               deployment/etcd-operator              Updating node "master-0" from revision 0 to 2 because node master-0 static pod not found
73m         Warning   ClusterMemberControllerUpdatingStatus   deployment/etcd-operator              Operation cannot be fulfilled on etcds.operator.openshift.io "cluster": the object has been modified; please apply your changes to the latest version and try again
72m         Normal    PodCreated                              deployment/etcd-operator              Created Pod/installer-2-master-0 -n openshift-etcd because it was missing
71m         Warning   ReportEtcdMembersErrorUpdatingStatus    deployment/etcd-operator              etcds.operator.openshift.io "cluster" not found
71m         Normal    RevisionTriggered                       deployment/etcd-operator              new revision 3 triggered by "configmap \"etcd-pod\" not found"
71m         Normal    OperatorLoglevelChange                  deployment/etcd-operator              Operator log level changed from "Debug" to "Normal"
71m         Normal    OperatorStatusChanged                   deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded changed from False to True ("EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is not ready\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is terminated: \"Error\" - \"d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379\\n#### attempt 0\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 1\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 2\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 3\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 4\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 5\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 6\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 7\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 8\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 9\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\ntimed out\"\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nEtcdMemberIPMigratorDegraded: context deadline exceeded")
71m         Normal    OperatorStatusChanged                   deployment/etcd-operator              Status for clusteroperator/etcd changed: Degraded message changed from "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is not ready\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is terminated: \"Error\" - \"d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379\\n#### attempt 0\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 1\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 2\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 3\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 4\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 5\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 6\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 7\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 8\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 9\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\ntimed out\"\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nEtcdMemberIPMigratorDegraded: context deadline exceeded" to "EtcdMembersControllerDegraded: node lister not synced\nBootstrapTeardownDegraded: node lister not synced\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is not ready\nStaticPodsDegraded: nodes/master-0 pods/etcd-master-0 container=\"etcd\" is terminated: \"Error\" - \"d5e42fbbde2f1d75, started, etcd-bootstrap, https://192.168.222.127:2380, https://192.168.222.127:2379\\n#### attempt 0\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 1\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 2\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 3\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 4\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 5\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 6\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 7\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 8\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\n#### attempt 9\\n      member={name=\\\"etcd-bootstrap\\\", peerURLs=[https://192.168.222.127:2380}, clientURLs=[https://192.168.222.127:2379]\\n      target=nil, err=<nil>\\n#### sleeping...\\ntimed out\"\nStaticPodsDegraded: pods \"etcd-master-2\" not found\nStaticPodsDegraded: pods \"etcd-master-1\" not found\nScriptControllerDegraded: \"configmap/etcd-pod\": missing env var values\nEtcdMemberIPMigratorDegraded: context deadline exceeded"

Comment 1 Sai Sindhur Malleni 2020-05-07 17:43:28 UTC

Created attachment 1686261 [details]
etcd-operator.log

Comment 3 Suresh Kolichala 2020-05-20 13:00:56 UTC

This problem looks very similar to another we are chasing on bare metal. Closing it as duplicate to 1814576.

*** This bug has been marked as a duplicate of bug 1814576 ***