Description of problem:
After deploying monitoring, Prometheus is unable to access etcd and the logs show x509 error.
Sounds similar to https://bugzilla.redhat.com/show_bug.cgi?id=1645417
Version-Release number of selected component (if applicable):
3.11
How reproducible:
1. Deploy Prometheus monitoring
2. Prometheus is unable to monitor etcd due to not having a secret
3. Create the kube-etcd-client-certs secret in the openshift-monitoring project
4. Prometheus does not detect and use this new secret and so still cannot monitor etcd
5. A temporary workaround is to manually add the secret in the GUI which allows the daemon set to be created -- but this does not persist
Actual results:
The Prometheus operator does not automatically create a secret to monitor etcd.
A manually added secret does not get noticed by Prometheus operator.
A manually added secret using the 'add this secret to an application' dialogue in the GUI works around this issue but does not persist.
This error is seen in Prometheus log:
level=error ts=2019-08-15T11:12:15.605123238Z caller=scrape.go:148 component="scrape manager" scrape_pool=openshift-monitoring/etcd/0 msg="Error creating HTTP client" err="unable to use specified CA cert /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client-ca.crt: open /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client-ca.crt: no such file or directory
Expected results:
1. Prometheus should create the correct secret for monitoring etcd or
2. Prometheus should detect that a secret for monitoring etcd has been created.
3. Also, a secret added in the GUI should persist.