Bug 1660778 - The configuration of caFile in servicemonitor object is incorrect.
Summary: The configuration of caFile in servicemonitor object is incorrect.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.1.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-19 08:38 UTC by Qiaoling Tang
Modified: 2019-06-04 10:41 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2019-06-04 10:41:27 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:41:32 UTC
Github openshift cluster-logging-operator pull 65 None None None 2018-12-19 15:40:05 UTC
Github openshift elasticsearch-operator pull 60 None None None 2019-01-10 14:44:54 UTC

Description Qiaoling Tang 2018-12-19 08:38:03 UTC
Description of problem:
The path of caFile in prometheus pod has been changed, but the servicemonitor object in openshift-logging not changed yet. Find error log in prometheus pod:

level=error ts=2018-12-19T06:59:55.79674926Z caller=scrape.go:148 component="scrape manager" scrape_pool=openshift-logging/monitor-elasticsearch-cluster/0 msg="Error creating HTTP client" err="unable to use specified CA cert /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt: open /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt: no such file or directory"

$ oc -n openshift-monitoring rsh -c prometheus prometheus-k8s-0
sh-4.2$ ls /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt
ls: cannot access /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt: No such file or directory
sh-4.2$ ls /etc/prometheus/configmaps/serving-certs-ca-bundle/
service-ca.crt
sh-4.2$ ls /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
ls: cannot access /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt: No such file or directory
sh-4.2$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt	namespace  token
sh-4.2$ exit

$ oc get servicemonitor -n openshift-logging -o yaml |grep tlsConfig -A 1 -B 5
    uid: c635a072-028d-11e9-940f-0e40ce4b124a
  spec:
    endpoints:
    - port: "24231"
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
--
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      path: /_prometheus/metrics
      port: restapi
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt


Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.alpha-2018-12-17-160013   True        False         4h        Cluster version is 4.0.0-0.alpha-2018-12-17-160013


How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Qiaoling Tang 2018-12-19 08:40:39 UTC
Error Path in Logging servicemonitor object
/etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

Correct Path prometheus:
/etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle//service-ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Comment 3 Qiaoling Tang 2019-01-10 03:13:03 UTC
The caFile path in fluentd has changed, but not changed in monitor-elasticsearch-cluster.
$ oc get servicemonitors.monitoring.coreos.com  -o yaml |grep tlsConfig -A 1 -B 5
    uid: b843b250-1484-11e9-b91a-0282b3a38dc0
  spec:
    endpoints:
    - port: "24231"
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
--
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      path: /_prometheus/metrics
      port: restapi
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/prometheus-serving-certs-ca-bundle/service-ca.crt


They all should be:  /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt

Comment 4 Qiaoling Tang 2019-01-14 02:33:17 UTC
The issue has been fixed, move bug to VERIFIED.

Comment 7 errata-xmlrpc 2019-06-04 10:41:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.