Description of problem: Tested with HTTPS_PROXY cluster on 4.2.0-0.nightly-2020-02-24-065701, all monitoring routes can not be accessed, and returns "500 Internal Error", checked, trusted user-ca-bundle cert is not added to grafana/prometheus/alertmanager, only in telemeter-client pod # oc get proxy/cluster -oyaml apiVersion: config.openshift.io/v1 kind: Proxy metadata: creationTimestamp: "2020-02-25T05:11:04Z" generation: 1 name: cluster resourceVersion: "430" selfLink: /apis/config.openshift.io/v1/proxies/cluster uid: 38cad270-578d-11ea-9ed1-fa163ee6fdc1 spec: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3128 httpsProxy: https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3130 noProxy: test.no-proxy.com trustedCA: name: user-ca-bundle status: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3128 httpsProxy: https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3130 noProxy: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,172.30.0.0/16,api-int.juzhao-bug.qe.devcluster.openshift.com,etcd-0.juzhao-bug.qe.devcluster.openshift.com,etcd-1.juzhao-bug.qe.devcluster.openshift.com,etcd-2.juzhao-bug.qe.devcluster.openshift.com,localhost,test.no-proxy.com ***************************************************** # oc -n openshift-monitoring exec -c prometheus-proxy prometheus-k8s-0 -- env | grep PROXY HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3128 HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3130 NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,172.30.0.0/16,api-int.juzhao-bug.qe.devcluster.openshift.com,etcd-0.juzhao-bug.qe.devcluster.openshift.com,etcd-1.juzhao-bug.qe.devcluster.openshift.com,etcd-2.juzhao-bug.qe.devcluster.openshift.com,localhost,test.no-proxy.com ***************************************************** # oc -n openshift-monitoring exec -c alertmanager-proxy alertmanager-main-0 -- env | grep PROXY HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3128 HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3130 NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,172.30.0.0/16,api-int.juzhao-bug.qe.devcluster.openshift.com,etcd-0.juzhao-bug.qe.devcluster.openshift.com,etcd-1.juzhao-bug.qe.devcluster.openshift.com,etcd-2.juzhao-bug.qe.devcluster.openshift.com,localhost,test.no-proxy.com ***************************************************** # oc -n openshift-monitoring exec -c grafana-proxy grafana-88777fff9-hcpr9 -- env | grep PROXY HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3130 NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,172.30.0.0/16,api-int.juzhao-bug.qe.devcluster.openshift.com,etcd-0.juzhao-bug.qe.devcluster.openshift.com,etcd-1.juzhao-bug.qe.devcluster.openshift.com,etcd-2.juzhao-bug.qe.devcluster.openshift.com,localhost,test.no-proxy.com HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.77.163:3128 ***************************************************** # oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus-proxy 2020/02/25 05:55:18 provider.go:573: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server 2020/02/25 05:55:18 provider.go:613: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server { "issuer": "https://oauth-openshift.apps.juzhao-bug.qe.devcluster.openshift.com", "authorization_endpoint": "https://oauth-openshift.apps.juzhao-bug.qe.devcluster.openshift.com/oauth/authorize", "token_endpoint": "https://oauth-openshift.apps.juzhao-bug.qe.devcluster.openshift.com/oauth/token", "scopes_supported": [ "user:check-access", "user:full", "user:info", "user:list-projects", "user:list-scoped-projects" ], "response_types_supported": [ "code", "token" ], "grant_types_supported": [ "authorization_code", "implicit" ], "code_challenge_methods_supported": [ "plain", "S256" ] } 2020/02/25 05:55:18 oauthproxy.go:645: error redeeming code (client:10.128.2.1:37820): Post https://oauth-openshift.apps.juzhao-bug.qe.devcluster.openshift.com/oauth/token: proxyconnect tcp: x509: certificate signed by unknown authority 2020/02/25 05:55:18 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error ***************************************************** # oc -n openshift-config get cm user-ca-bundle -oyaml apiVersion: v1 data: ca-bundle.crt: | -----BEGIN CERTIFICATE----- MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x EDAOBgNVBAgMB0JlaWppbmcxEDAOBgNVBAcMB0JlaWppbmcxDDAKBgNVBAoMA09D UDEPMA0GA1UECwwGT0NQLVFFMRcwFQYDVQQDDA5PQ1AtUUUtUk9PVC1DQTAeFw0x OTA4MTgwNjA4MzRaFw0yOTA4MTUwNjA4MzRaMF4xCzAJBgNVBAYTAkNOMRAwDgYD VQQIDAdCZWlqaW5nMQwwCgYDVQQKDANPQ1AxFTATBgNVBAsMDEluc3RhbGxlci1R RTEYMBYGA1UEAwwPSW5zdGFsbGVyLVFFLUNBMIICIjANBgkqhkiG9w0BAQEFAAOC Ag8AMIICCgKCAgEAwt0MujtrS6uPOx9pV71W5o0Nk9a6Fe4bSojyyOJw1SmDihaC AvxrWK3NHaqYV8cqQWLB1ZXtw8LF74BK98/b94PvauqgTn3Kg+Vcqnq3JlpyrgKN n5g4ORYScQXlyN/Kzn98cv07qHn1MhwZt8W8lYI9m6z2un0VyPkr8UgSmvDo0cx0 zwjB5Q7zCvXcoc1IQFa3JkYH4Z6Ccz9FNYnDRtoqu8K3SiWid50WEXcpycMLCSwb SVSDAsUR5wwA4aTgW7s32Fdd4fAtNcnfZ2AnLTwyJBZoPeoa5npvmpCr8khLyDdW Y9rWDfaKXhB++Ou27FDE6NLWQK/FPMVNPIr+P3xPbHIDlwzWq0eSK8SMsiOZrI9N dzMNGtcxv3sfxMYqKhnl3HrZbXbM1ouD9lsv5zGCAIdrnmZoMRI9NTjBatOevZXQ ojby2XQzNDX1ouQK4gSTi9q3aa1e8WQfiLbaNPxAU9FlLqS7J16nFsTsWQ6Qt6iN yEFaw3pYWeZk6sacGQECvmfrbaHxlI63rQUI3mRxs8mZqb3zJapcbNtUlimEAsqE 1oj/Tv3oVQKei2MpQHctenJqOZGC0Q/iWeRALD9E656MqbIt5dudEnx56Nq8av4r sad+OquDKFB/EnQ69VViYs9s6Ck426bqX5dx6T0Y0Tgk0WcnR5aPO+YrEtUCAwEA AaNmMGQwHQYDVR0OBBYEFJhUiRBfCjzfjHxoPLwYEwz5jHuuMB8GA1UdIwQYMBaA FG5nokgYqmIIwaW7blM6wHVIQwBIMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0P AQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBhcO+rA1blMP7SKt3/qzqsX5di BRxqOWqlmpKDgmC9rJts76t/PEodI2XNUVnKtybQD7Fh768b4fo0WO/evWUxs2LM 4d7jQp5KTqEPhv6oKlrTp9fzw3BGwdnzZSPk6L8ahZvyr0i7Hls9oe5Pvhy5F87e qWt/SuDMCztYR3gs78IxBYMv4BPEuCeLsvLlPFW4vl+4lpGjOGcS8GbwwZIwq5X4 LIdkk00NAMQ6Nmztoc+k/EVnj7O/bj66FY4WZFYUgnKUMlJ33UZy+Uao2GKUAM8j znFOl8fHgLYlcHsRYyLWeMGmOk0ukN06AvygnWh0UVBQCRrmTPNsShK+PlRyHmFW Zw4TDuPOqEwLx1VcmlEbLbpgc4f4GUWKGegaLHUltfwTwlb/6m1J4HomiYrBhdLJ LDReBo7dNYr7mpGPfZIMRdmywz6w10F1zTKe2F1KHb7mR7tyORaZ7NcAtmQmuxDF T8sUTrIop4GaQMZnNTPImtPGt23zsNTXUY93IeISJ6eUDKlnDgzYJDQ3pnKWbWHz wdWcyjh0Ojh/snItIm6/h1+CQ/FRlnt3+LRP9GxvWHbn1+sS51Kb979m/R0W7Djt y4p+AwCHpLwi9sU17Lg1JafgJVFB9Tu2wz/DIocfzdpP+7MUrqTkeDmN0p+Ia1Y9 bTSegOgySxp2uzPJqg== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIFuDCCA6CgAwIBAgIJAJk39xzKHHf9MA0GCSqGSIb3DQEBCwUAMGkxCzAJBgNV BAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMRAwDgYDVQQHDAdCZWlqaW5nMQwwCgYD VQQKDANPQ1AxDzANBgNVBAsMBk9DUC1RRTEXMBUGA1UEAwwOT0NQLVFFLVJPT1Qt Q0EwHhcNMTkwODE4MDYwNzU4WhcNMzkwODEzMDYwNzU4WjBpMQswCQYDVQQGEwJD TjEQMA4GA1UECAwHQmVpamluZzEQMA4GA1UEBwwHQmVpamluZzEMMAoGA1UECgwD T0NQMQ8wDQYDVQQLDAZPQ1AtUUUxFzAVBgNVBAMMDk9DUC1RRS1ST09ULUNBMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA338oV6IIvllZpr/lWOjMMVZq 4Smm0qA6BOe3ezZlr2LU5TLsgZeY+Oa1YtwXSAka8rRnuuqNa6gZEHGdL1SHTynB rEyq05KErChLabRVYb9aotQHt1+G1GG2Mi11QZ4Pdgsfmrs8NC05703C5V4kEL+q NXG88O3J54ySsKp+aD4xvOtZ0uXcVdjAo347/CJEm/2HF9C/uIR8ktJ43ZQPq55c tgsJjjY/UBSmOOhDsTfRzv9DVrcWuZYW0ZztG7gfC3d2i2l7dLhaAr76kzZ68aH2 402ghE1Xh9zDlmWugfqOyT/v6RsE7gL/Dkkuk27Eau3jyRdWVIJroqK2Sd/yJcrQ DiG1wAzwb7JVlPi5lkQBrWXti+qgm415+Xfcc9KRZP3hv3tbGVuKmNxONpGjbrMw GKV2EMWGnpdKepQ0STWb9SC916iNXO9ffCsPlqgEoV1ONiNfvU9G3cCcRcc1yjtF 8zbMcqmtsvl+AC1RfmM4n8TesSx56vk/obNsUljtU1/FGQIKRlamey4r/dKDR8kJ oyDibv7dUGm5pX5/L7bahRb7LoVg0MbV9bGlqL+hpCbjIO1rouMyy3qu3z+NMGh7 nzVYULulOjdbVw5u14O4VeonavWByyCFUMK4JKqfUOPNjjS7OEXue1HoCy9LBjIv qfPUdeulyX0OtbZ8EhECAwEAAaNjMGEwHQYDVR0OBBYEFG5nokgYqmIIwaW7blM6 wHVIQwBIMB8GA1UdIwQYMBaAFG5nokgYqmIIwaW7blM6wHVIQwBIMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQAcnBrb Cde2jE+iumzlN3TNm6nOvMnomIrMupBInuWI0GvA9rGjv8SC8ZAjfx/fZOY28uLx ACiZqKWQT0YARjKCgOSe0RxTG+vpNH6E8FpTEiVIq/N+rgdHCZUJiWoY7BA1FNNq 3UTlqV6RM+RqsVIptu8lk7fVDehng+zQzYYs4ZV6bSLjBQG3yBUBN1lYnFWe3pnS WmLuw22Riuunc5MVdH97modji1UDzQHDbYy0FXt8gLM8DRPIrOe039XO1lO+eWWM /NI7sZBU6bSotDh3aTLnHIyJdJ0dnh+/wMIK6h5au/7BMV1oK4JsSmpNCmzP+s3O cpNINYhkBRqFViA72D/Vim/meP2Q4J/dKsT2JbprY7X/XIYd1+aS48QAyusat2Gn KJ1JQNOoYHGijz8bYHm5JVytMIKU5LJ/Rp9SgK3d0ByqmJR76alzyRdUKa3Pmsw3 Beq8GQSAdjlyIB6C1FpG7XD4ySz1EjGEcOXiGiEi8l9wjDgLtA20U9ALaMcEdODY K8zhyirrdXdV8XHBAE7QBkzcuQAVc9iyTNoqCfJBtvl2HYpH2XoRhxP0rX9NtAYE Gc+Yc4Tgf2HAERrwj0B6AfWQaDfcjAJtQ0xorONJJpEZpItV8Cl5dSeOtX7howTB BvBHcmyVbaW7PGNBmIM1FBKwi/fBJoawSJlslA== -----END CERTIFICATE----- kind: ConfigMap metadata: creationTimestamp: "2020-02-25T05:10:56Z" name: user-ca-bundle namespace: openshift-config resourceVersion: "239" selfLink: /api/v1/namespaces/openshift-config/configmaps/user-ca-bundle uid: 34228dd0-578d-11ea-9ed1-fa163ee6fdc1 ***************************************************** check if the user-ca-bundle cert is in alertmanager/prometheus/grafana, use the first line "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" to check # oc -n openshift-monitoring exec -c alertmanager-proxy alertmanager-main-0 -- cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" no return # oc -n openshift-monitoring exec -c prometheus-proxy prometheus-k8s-0 -- cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" no return # oc -n openshift-monitoring exec -c grafana-proxy grafana-88777fff9-hcpr9 -- cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" no return only telemeter-client include the user-ca-bundle cert # oc -n openshift-monitoring exec -c telemeter-client telemeter-client-6cf757d694-vk6g9 -- cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x ***************************************************** Version-Release number of selected component (if applicable): # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.2.0-0.nightly-2020-02-24-065701 True False 18m Cluster version is 4.2.0-0.nightly-2020-02-24-065701 How reproducible: Always Steps to Reproduce: 1. Login HTTPS_PROXY 4.2 monitoring UIs 2. 3. Actual results: returns "500 Internal Error" Expected results: no error Additional info:
Created attachment 1665567 [details] monitoring dump file, see logs here
Created attachment 1665587 [details] alertmanager file
the user-ca-bundle cert is in alertmanager-trusted-ca-bundle configmap file # oc -n openshift-monitoring get cm alertmanager-trusted-ca-bundle -oyaml | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x # oc -n openshift-monitoring get cm alertmanager-trusted-ca-bundle-6srk7hbuke4sh -oyaml | grep "MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x" MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x
Assigned this to 4.5. We can work out where we need to backport to after resolving the issue.
Hi Junqi Could you reproduce the problem on a test cluster so that I can take a closer look? Thanks in advance /Alex
Checked, 4.2 HTTPS_PROXY,there is not /etc/pki/ca-trust/extracted/pem/ for alertmanager-trusted-ca-bundle, it is /etc/pki/alertmanager-ca-bundle/; no /etc/pki/ca-trust/extracted/pem/ for prometheus-trusted-ca-bundle, it is /etc/pki/prometheus-ca-bundle/; and grafana-trusted-ca-bundle is mount for grafana container, which is wrong, should be for grafana-proxy container 4.2 HTTPS_PROXY - mountPath: /etc/pki/alertmanager-ca-bundle/ name: alertmanager-trusted-ca-bundle readOnly: true - mountPath: /etc/pki/prometheus-ca-bundle/ name: prometheus-trusted-ca-bundle readOnly: true name: grafana .... - mountPath: /etc/pki/ca-trust/extracted/pem/ name: grafana-trusted-ca-bundle readOnly: true 4.3 HTTPS_PROXY - mountPath: /etc/pki/ca-trust/extracted/pem/ name: alertmanager-trusted-ca-bundle readOnly: true - mountPath: /etc/pki/ca-trust/extracted/pem/ name: prometheus-trusted-ca-bundle readOnly: true name: grafana-proxy ... - mountPath: /etc/pki/ca-trust/extracted/pem/ name: grafana-trusted-ca-bundle readOnly: true
Created attachment 1671904 [details] 42 HTTPS_PROXY info
Created attachment 1671905 [details] 43 HTTPS_PROXY info
checked on 4.2.0-0.nightly-2020-04-20-085107 HTTPS_PROXY cluster, grafana-trusted-ca-bundle is under grafana-proxy container and grafana UI is accessible
Created attachment 1681027 [details] grafana pod's file on 4.2.0-0.nightly-2020-04-20-085107 HTTPS_PROXY cluster
(In reply to Junqi Zhao from comment #31) > checked on 4.2.0-0.nightly-2020-04-20-085107 HTTPS_PROXY cluster, > grafana-trusted-ca-bundle is under grafana-proxy container and grafana UI is > accessible alertmanager/prometheus UI are still 500 error
The only thing we have left is that the route links to alertmanager and prometheus UI cannot work as the trusted ca bundle cert is not mounted into the oauthproxy containers in the correct path where oauthproxy expects. The solution is not as straightforward as we use CustomResources to create those objects and there is a known bug for ConfigMap mounting in that kubernetes version of openshift 4.2, this is why it just works in 4.3, I asked if the bug that was fixed in later versions will get backported to 4.2, still waiting for the answer. There is a workaround for this for customers, the console alerting and monitoring pages work just fine and users can use port-forward to access the alertmanager and prometheus UIs. As we cannot solve this from our side, we are closing this issue as there are known (two) workarounds and grafana was fixed as part of this bugzilla already. Our stack is fully functional. @sur @junqi should we close as won't fix or not a bug, because we did fix the grafana issue with this, wdyt?