Bug 1750433

Description of problem:
Deploying a cluster using https://github.com/openshift-kni/install-scripts (which BTW, uses the internal ocp/release:4.2 image https://github.com/openshift-kni/install-scripts/blob/master/OpenShift/03_create_cluster.sh#L7) it seems the etcd cluster doesn't form properly as one of the etcd-member pods is in crashloopback state.
The installer finish successfully tho.

Version-Release number of selected component (if applicable):
$ oc version
Client Version: openshift-clients-4.2.0-201909081401
Server Version: 4.2.0-0.nightly-2019-09-09-073137
Kubernetes Version: v1.14.6+9d7f0a8


How reproducible:
See https://github.com/openshift-kni/install-scripts/issues/100 (the title seems to be misleading)


Steps to Reproduce:
1. Deploy an OCP4 cluster with install-scripts or dev-scripts (https://github.com/openshift-metal3/dev-scripts)
2. Verify the etcd-member pods are ok
3.

Actual results:
1/3 is not:

$ oc get pods -n openshift-etcd
NAME                                                     READY   STATUS             RESTARTS   AGE
etcd-member-kni1-master-0.env.mydomain.example.com   2/2     Running            0          87m
etcd-member-kni1-master-1.env.mydomain.example.com   1/2     CrashLoopBackOff   17         87m
etcd-member-kni1-master-2.env.mydomain.example.com   2/2     Running            0          87m

$ oc logs etcd-member-kni1-master-1.env.mydomain.example.com -n openshift-etcd
Error from server (BadRequest): a container name must be specified for pod etcd-member-kni1-master-1.env.mydomain.example.com, choose one of: [etcd-member etcd-metrics] or one of the init containers: [discovery certs]

$ oc logs etcd-member-kni1-master-1.env.mydomain.example.com -c etcd-member -n openshift-etcd
/bin/sh: line 3: /run/etcd/environment: Permission denied

$ for node in $(oc get nodes -o jsonpath="{.items[*].metadata.name}"); do ssh core@${nod
e} sudo cat /run/etcd/environment; ssh core@${node} sudo ls -lZ /run/etcd/environment; done

export ETCD_DISCOVERY_SRV=kni1.env.mydomain.example.com
ETCD_WILDCARD_DNS_NAME=*.kni1.env.mydomain.example.com
ETCD_IPV4_ADDRESS=10.19.138.11
ETCD_DNS_NAME=etcd-0.kni1.env.mydomain.example.com
-rw-r--r--. 1 root root system_u:object_r:container_var_run_t:s0 205 Sep  9 11:29 /run/etcd/environment

export ETCD_DISCOVERY_SRV=kni1.env.mydomain.example.com
ETCD_IPV4_ADDRESS=10.19.138.12
ETCD_DNS_NAME=etcd-1.kni1.env.mydomain.example.com
ETCD_WILDCARD_DNS_NAME=*.kni1.env.mydomain.example.com
-rw-r--r--. 1 root root system_u:object_r:container_var_run_t:s0 205 Sep  9 11:29 /run/etcd/environment

export ETCD_DISCOVERY_SRV=kni1.env.mydomain.example.com
ETCD_WILDCARD_DNS_NAME=*.kni1.env.mydomain.example.com
ETCD_IPV4_ADDRESS=10.19.138.13
ETCD_DNS_NAME=etcd-2.kni1.env.mydomain.example.com
-rw-r--r--. 1 root root system_u:object_r:container_var_run_t:s0 205 Sep  9 11:28 /run/etcd/environment

Expected results:
3/3 pods up

Additional info:

I've just moved the etcd-member static pod definition in the affected host (to simulate a oc delete but for the static pod) and it seems to fix it...

$ ssh core.mydomain.example.com sudo mv /etc/kubernetes/manifests/etcd-member.yaml /root/
$ ssh core.mydomain.example.com sudo mv /root/etcd-member.yaml /etc/kubernetes/manifests/etcd-member.yaml

$ oc get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
etcd-member-kni1-master-0.env.mydomain.example.com   2/2     Running   2          129m
etcd-member-kni1-master-1.env.mydomain.example.com   2/2     Running   28         23m
etcd-member-kni1-master-2.env.mydomain.example.com   2/2     Running   0          129m