Bug 1788477

Summary: proxy user-ca-bundle did not present in grafana pod /etc/pki/tls/certs/ca-bundle.crt
Product: OpenShift Container Platform Reporter: Muhammad Aizuddin Zali <mzali>
Component: MonitoringAssignee: Pawel Krupa <pkrupa>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, mzali, pkrupa, scuppett, surbania
Target Milestone: ---   
Target Release: 4.2.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-22 10:46:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1776085    
Bug Blocks:    

Description Muhammad Aizuddin Zali 2020-01-07 10:19:21 UTC 
Description of problem:
When replacing default ingress certificate with custom organization root CA, added root CA into proxies/config. Grafana pod did not take the root CA into it pod CA bundle. Hence grafana proxy POST call  auth URL returned below error:

2020/01/07 10:14:11 oauthproxy.go:645: error redeeming code (client:10.9.0.1:43806): Post https://oauth-openshift.apps.ocp4.local.bytewise.my/oauth/token: x509: certificate signed by unknown authority
2020/01/07 10:14:11 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error


Version-Release number of selected component (if applicable):
4.1.12

How reproducible:

Follow this guide[1] to add cluster wide new custom root CA and then replace default ingress certificate from here[2]. Login to the cluster and open up Grafana dashboard will shown above error.


[1]: https://docs.openshift.com/container-platform/4.2/networking/enable-cluster-wide-proxy.html
[2]: https://docs.openshift.com/container-platform/4.2/authentication/certificates/replacing-default-ingress-certificate.html

Steps to Reproduce:
1.
2.
3.

Actual results:
Grafana CA trust bundle did not include proxies certificate for trusting custom root CA.



Expected results:
Grafana CA trust bundle should include proxies certificate and trust the custom root CA.

Additional info:
Comment 1 Muhammad Aizuddin Zali 2020-01-07 10:23:44 UTC 
1. The custom root CA:

mzali  ~  cat  CA-Auth/rootCA.crt  | head -n 2
-----BEGIN CERTIFICATE-----
MIIFxTCCA62gAwIBAgIUIQeGpl00JG5PC1Db0xrNpW9aiu0wDQYJKoZIhvcNAQEL

2. The trust bundle does not include the root CA:

mzali  ~  oc rsh grafana-6cc69d4d4b-5xc7b 
Defaulting container name to grafana.
Use 'oc describe pod/grafana-6cc69d4d4b-5xc7b -n openshift-monitoring' to see all of the containers in this pod.
sh-4.2$ cat /etc/pki/tls/certs/ca-bundle.crt | grep MIIFxTCCA62gAwIBAgIUIQeGpl00JG5PC1Db0xrNpW9aiu0wDQYJKoZIhvcNAQEL
sh-4.2$
Comment 2 Stephen Cuppett 2020-01-07 13:28:45 UTC 
Setting target to active development branch (4.4). Fixes (if any) will result in cloned BZs for backports (where required/requested) for prior versions.
Comment 11 errata-xmlrpc 2020-01-22 10:46:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0107