Created attachment 1524975 [details]
etcd target is 0/0 UP

Description of problem:
Cloned from https://jira.coreos.com/browse/MON-532

after enable etcd monitoring, etcd target is 0/0 UP

Steps:

1. Copy the etcd crt and key credentials files from the master node /etc/kubernetes/static-pod-resources/etcd-member directory to the local machine
$ls /etc/kubernetes/static-pod-resources/etcd-member

ca.crt system:etcd-peer:juzhao-etcd-1.qe.devcluster.openshift.com.crt system:etcd-server:juzhao-etcd-1.qe.devcluster.openshift.com.key
etcd-cert-secret.yaml system:etcd-peer:juzhao-etcd-1.qe.devcluster.openshift.com.key
root-ca.crt system:etcd-server:juzhao-etcd-1.qe.devcluster.openshift.com.crt

in this case, I copied system:etcd-peer:juzhao-etcd-1.qe.devcluster.openshift.com.crt as ca.crt for short
copied ssystem:etcd-peer:juzhao-etcd-1.qe.devcluster.openshift.com.key as ca.key for short

2. Create the openssl.cnf file with these contents:
[ req ]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[ req_distinguished_name ]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, keyEncipherment, digitalSignature
extendedKeyUsage=serverAuth, clientAuth

3. Generate the etcd.key private key file:
$openssl genrsa -out etcd.key 2048

4. Generate the etcd.csr certificate signing request file:
$openssl req -new -key etcd.key -out etcd.csr -subj "/CN=etcd" -config openssl.cnf

5. Generate the etcd.crt certificate file:
$openssl x509 -req -in etcd.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out etcd.crt -days 365 -extensions v3_req -extfile openssl.cnf

6. Put the credentials into format used by OpenShift, this creates file etcd-cert-secret.yaml.

cat <<-EOF > etcd-cert-secret.yaml
apiVersion: v1
data:
etcd-client-ca.crt: "$(cat ca.crt | base64 --wrap=0)"
etcd-client.crt: "$(cat etcd.crt | base64 --wrap=0)"
etcd-client.key: "$(cat etcd.key | base64 --wrap=0)"
kind: Secret
metadata:
name: kube-etcd-client-certs
namespace: openshift-monitoring
type: Opaque
EOF

7. Copy the file etcd-cert-secret.yaml to master and apply the credentials file to the cluster:
$oc apply -f etcd-cert-secret.yaml Secret/kube-etcd-client-certs is created

8. Configure ectd monitoring
$oc -n openshift-monitoring create configmap cluster-monitoring-config.yaml

cluster-monitoring-config.yaml content:

apiVersion: v1
data:
  config.yaml: |
    telemeterClient:
      enabled: false
    etcd:
      enabled: true
      targets:
        selector:
          k8s-app: etcd
kind: ConfigMap

selector is got by $ oc -n kube-system get po -oyaml | grep -i label -A 3

 



Version-Release number of selected component (if applicable):
payload:
image: registry.svc.ci.openshift.org/ocp/release@sha256:aa2c0365957e6c7733fc3dfd21d9f06b95e7664b325620a19becfc5a665caf68

 

images:

cluster-monitoring-operator: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03f03ef1202b0cd018bad3fbf21e2a3c4a2f30e23200cb61072441cb9e59a966
prometheus-node-exporter: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0410b28aeaff7e091dcbdb46a44f42aa7c2e441e6748c18601ba36a0019ac466
k8s-prometheus-adapter: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d14ced707696de3dc0ca7fc585a5240c36e75507639f01c805bef633fbe056c
prometheus-operator: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ac9a2c7980da5d32eb7700a5a57abd35e8f78aa2f662dbf2934fa566cacb4ed
prometheus-config-reloader: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56484d369661fd5b44bf3744d61c6f321594ddbb9a10e861030fc2d87dd63318
kube-rbac-proxy: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:701eb093944140bbb62fa6b966e3285f10671ae824fd1082ee69ae18386fd0a2
prom-label-proxy: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76264395fb1fb3d9053f64b5e4138735f1e53c9cc203ba7c5bfa4e55a1d27c78
prometheus: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f90d53707b67dde283c57549f8ad51b1f7e19134bfacff6266907a404c5eb09
kube-state-metrics: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:855c1f1f53a4c09add1f86f2fcfb2dffa256fd46d3e7d3cf4314a7bbce175f3e
prometheus-alertmanager: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8b1c8c96e17e5243310a4457eec7a8eded2320cf0a93ee902fc39000a853168
configmap-reloader: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba8b77ca06580db7e59e7ef452c233c2f0cb9f2c9ad9729255a05e3952675a5f
quay.io/openshift/origin-grafana:latest
quay.io/openshift/origin-oauth-proxy:latest

How reproducible:


Steps to Reproduce:
1. See Description part
2.
3.

Actual results:
etcd target is 0/0 UP

Expected results:
etcd target should be UP

Additional info: