Bug 1806854
| Summary: | trusted user-ca-bundle cert is not added to grafana/prometheus/alertmanager | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||||||||||
| Component: | Monitoring | Assignee: | Lili Cosic <lcosic> | ||||||||||||
| Status: | CLOSED WONTFIX | QA Contact: | Junqi Zhao <juzhao> | ||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||
| Priority: | high | ||||||||||||||
| Version: | 4.2.z | CC: | aconstan, adeshpan, alegrand, anpicker, aos-bugs, bbennett, cshulman, erooth, gparente, kakkoyun, lcosic, mfojtik, mloibl, pbertera, pkrupa, surbania | ||||||||||||
| Target Milestone: | --- | ||||||||||||||
| Target Release: | 4.5.0 | ||||||||||||||
| Hardware: | Unspecified | ||||||||||||||
| OS: | Unspecified | ||||||||||||||
| Whiteboard: | groom | ||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | Environment: | ||||||||||||||
| Last Closed: | 2020-04-27 07:35:01 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Bug Depends On: | |||||||||||||||
| Bug Blocks: | 1803957 | ||||||||||||||
| Attachments: |
|
||||||||||||||
|
Description
Junqi Zhao
2020-02-25 07:35:40 UTC
Created attachment 1665567 [details]
monitoring dump file, see logs here
Created attachment 1665587 [details]
alertmanager file
the user-ca-bundle cert is in alertmanager-trusted-ca-bundle configmap file
# oc -n openshift-monitoring get cm alertmanager-trusted-ca-bundle -oyaml | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x"
MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x
# oc -n openshift-monitoring get cm alertmanager-trusted-ca-bundle-6srk7hbuke4sh -oyaml | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x"
MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x
Assigned this to 4.5. We can work out where we need to backport to after resolving the issue. Hi Junqi Could you reproduce the problem on a test cluster so that I can take a closer look? Thanks in advance /Alex Checked, 4.2 HTTPS_PROXY,there is not /etc/pki/ca-trust/extracted/pem/ for alertmanager-trusted-ca-bundle, it is /etc/pki/alertmanager-ca-bundle/;
no /etc/pki/ca-trust/extracted/pem/ for prometheus-trusted-ca-bundle, it is /etc/pki/prometheus-ca-bundle/; and grafana-trusted-ca-bundle is mount for grafana container, which is wrong, should be for grafana-proxy container
4.2 HTTPS_PROXY
- mountPath: /etc/pki/alertmanager-ca-bundle/
name: alertmanager-trusted-ca-bundle
readOnly: true
- mountPath: /etc/pki/prometheus-ca-bundle/
name: prometheus-trusted-ca-bundle
readOnly: true
name: grafana
....
- mountPath: /etc/pki/ca-trust/extracted/pem/
name: grafana-trusted-ca-bundle
readOnly: true
4.3 HTTPS_PROXY
- mountPath: /etc/pki/ca-trust/extracted/pem/
name: alertmanager-trusted-ca-bundle
readOnly: true
- mountPath: /etc/pki/ca-trust/extracted/pem/
name: prometheus-trusted-ca-bundle
readOnly: true
name: grafana-proxy
...
- mountPath: /etc/pki/ca-trust/extracted/pem/
name: grafana-trusted-ca-bundle
readOnly: true
Created attachment 1671904 [details]
42 HTTPS_PROXY info
Created attachment 1671905 [details]
43 HTTPS_PROXY info
checked on 4.2.0-0.nightly-2020-04-20-085107 HTTPS_PROXY cluster, grafana-trusted-ca-bundle is under grafana-proxy container and grafana UI is accessible Created attachment 1681027 [details]
grafana pod's file on 4.2.0-0.nightly-2020-04-20-085107 HTTPS_PROXY cluster
(In reply to Junqi Zhao from comment #31) > checked on 4.2.0-0.nightly-2020-04-20-085107 HTTPS_PROXY cluster, > grafana-trusted-ca-bundle is under grafana-proxy container and grafana UI is > accessible alertmanager/prometheus UI are still 500 error The only thing we have left is that the route links to alertmanager and prometheus UI cannot work as the trusted ca bundle cert is not mounted into the oauthproxy containers in the correct path where oauthproxy expects. The solution is not as straightforward as we use CustomResources to create those objects and there is a known bug for ConfigMap mounting in that kubernetes version of openshift 4.2, this is why it just works in 4.3, I asked if the bug that was fixed in later versions will get backported to 4.2, still waiting for the answer. There is a workaround for this for customers, the console alerting and monitoring pages work just fine and users can use port-forward to access the alertmanager and prometheus UIs. As we cannot solve this from our side, we are closing this issue as there are known (two) workarounds and grafana was fixed as part of this bugzilla already. Our stack is fully functional. @sur @junqi should we close as won't fix or not a bug, because we did fix the grafana issue with this, wdyt? |