Bug 1660778

Summary: The configuration of caFile in servicemonitor object is incorrect.
Product: OpenShift Container Platform Reporter: Qiaoling Tang <qitang>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, rmeggins, surbania
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:41:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Qiaoling Tang 2018-12-19 08:38:03 UTC
Description of problem:
The path of caFile in prometheus pod has been changed, but the servicemonitor object in openshift-logging not changed yet. Find error log in prometheus pod:

level=error ts=2018-12-19T06:59:55.79674926Z caller=scrape.go:148 component="scrape manager" scrape_pool=openshift-logging/monitor-elasticsearch-cluster/0 msg="Error creating HTTP client" err="unable to use specified CA cert /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt: open /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt: no such file or directory"

$ oc -n openshift-monitoring rsh -c prometheus prometheus-k8s-0
sh-4.2$ ls /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt
ls: cannot access /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt: No such file or directory
sh-4.2$ ls /etc/prometheus/configmaps/serving-certs-ca-bundle/
service-ca.crt
sh-4.2$ ls /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
ls: cannot access /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt: No such file or directory
sh-4.2$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt	namespace  token
sh-4.2$ exit

$ oc get servicemonitor -n openshift-logging -o yaml |grep tlsConfig -A 1 -B 5
    uid: c635a072-028d-11e9-940f-0e40ce4b124a
  spec:
    endpoints:
    - port: "24231"
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
--
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      path: /_prometheus/metrics
      port: restapi
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt


Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.alpha-2018-12-17-160013   True        False         4h        Cluster version is 4.0.0-0.alpha-2018-12-17-160013


How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Qiaoling Tang 2018-12-19 08:40:39 UTC
Error Path in Logging servicemonitor object
/etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

Correct Path prometheus:
/etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle//service-ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Comment 3 Qiaoling Tang 2019-01-10 03:13:03 UTC
The caFile path in fluentd has changed, but not changed in monitor-elasticsearch-cluster.
$ oc get servicemonitors.monitoring.coreos.com  -o yaml |grep tlsConfig -A 1 -B 5
    uid: b843b250-1484-11e9-b91a-0282b3a38dc0
  spec:
    endpoints:
    - port: "24231"
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
--
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      path: /_prometheus/metrics
      port: restapi
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt


They all should be:  /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt

Comment 4 Qiaoling Tang 2019-01-14 02:33:17 UTC
The issue has been fixed, move bug to VERIFIED.

Comment 7 errata-xmlrpc 2019-06-04 10:41:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758