Created attachment 1680730 [details] Authentication "500 Internal Error" Created attachment 1680730 [details] Authentication "500 Internal Error" Description of problem: Authentication "500 Internal Error" for all monitoring routes # kubectl -n openshift-monitoring get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD alertmanager-main alertmanager-main-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com alertmanager-main web reencrypt/Redirect None grafana grafana-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com grafana https reencrypt/Redirect None prometheus-k8s prometheus-k8s-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com prometheus-k8s web reencrypt/Redirect None thanos-querier thanos-querier-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com thanos-querier web reencrypt/Redirect None take prometheus route as an example, the UI shows 500 Internal Error, also see the error in container logs # oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus-proxy 2020/04/22 01:49:43 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s 2020/04/22 01:49:43 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token 2020/04/22 01:49:43 provider.go:312: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates. 2020/04/22 01:49:43 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/" 2020/04/22 01:49:43 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics" 2020/04/22 01:49:43 oauthproxy.go:227: OAuthProxy configured for Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s 2020/04/22 01:49:43 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled 2020/04/22 01:49:43 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth 2020/04/22 01:49:43 http.go:107: HTTPS: listening on [::]:9091 I0422 01:49:43.260202 1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key 2020/04/22 02:55:11 provider.go:394: authorizer reason: 2020/04/22 02:55:12 provider.go:394: authorizer reason: 2020/04/22 02:55:13 provider.go:575: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server 2020/04/22 02:55:13 provider.go:615: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server { "issuer": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com", "authorization_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/authorize", "token_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/token", "scopes_supported": [ "user:check-access", "user:full", "user:info", "user:list-projects", "user:list-scoped-projects" ], "response_types_supported": [ "code", "token" ], "grant_types_supported": [ "authorization_code", "implicit" ], "code_challenge_methods_supported": [ "plain", "S256" ] } 2020/04/22 02:55:18 provider.go:575: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server 2020/04/22 02:55:18 provider.go:615: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server { "issuer": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com", "authorization_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/authorize", "token_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/token", "scopes_supported": [ "user:check-access", "user:full", "user:info", "user:list-projects", "user:list-scoped-projects" ], "response_types_supported": [ "code", "token" ], "grant_types_supported": [ "authorization_code", "implicit" ], "code_challenge_methods_supported": [ "plain", "S256" ] } 2020/04/22 02:55:18 oauthproxy.go:645: error redeeming code (client:10.128.2.3:55456): Post https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/token: x509: certificate signed by unknown authority 2020/04/22 02:55:18 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error # openssl crl2pkcs7 -nocrl -certfile <(kubectl -n openshift-monitoring get secret prometheus-k8s-tls -o jsonpath='{.data.tls\.crt}' | base64 -d) | openssl pkcs7 -print_certs -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 5679961306105448820 (0x4ed346b6d4610974) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=openshift-service-serving-signer@1587519388 Validity Not Before: Apr 22 01:48:08 2020 GMT Not After : Apr 22 01:48:09 2022 GMT Subject: CN=prometheus-k8s.openshift-monitoring.svc Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:d3:4e:1a:ca:63:6b:f2:ad:1f:e0:17:f3:d1:21: 66:e8:37:d2:b1:d7:7a:ee:91:36:f2:ea:04:44:45: 23:91:a6:29:42:aa:b3:90:55:33:22:dc:72:3b:56: 9e:33:5d:13:fb:6c:5c:68:bd:76:1f:06:bf:35:f2: 33:e6:7d:67:d5:58:89:32:cf:ee:7b:af:03:fa:f1: 0b:a6:ff:70:24:c5:83:ed:ab:de:f7:b1:ee:44:b9: 8e:a8:ba:6a:d2:54:17:ba:5c:ba:64:8c:d6:a5:a5: b9:86:f9:f1:36:d8:c5:83:36:57:10:5d:b3:65:8a: 9b:99:57:7d:28:1c:92:3e:c5:74:66:ab:15:09:4d: e9:78:5a:d4:a1:0f:54:85:a8:3e:45:e6:d3:02:83: e5:31:96:17:4e:7c:61:e0:88:32:c7:25:05:2c:6f: ed:37:2b:8c:e1:68:cf:ad:05:7a:30:f2:d8:9c:ae: 3d:63:54:88:29:a1:25:d7:3a:1b:75:fd:b8:ce:c3: 47:85:8b:cf:b0:9f:fc:e8:ad:66:1e:b5:56:3c:a5: bb:27:07:fd:25:d3:27:c9:04:af:7a:29:1d:48:c1: db:21:ba:83:6b:2b:cb:01:fa:85:b5:08:05:6e:40: ac:94:f3:27:7f:64:42:6b:90:73:d4:b8:61:fb:50: 6e:6f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: 5B:11:44:7A:AC:BE:F0:6C:E7:B6:9F:60:80:FE:D2:C0:4E:4C:33:9E X509v3 Authority Key Identifier: keyid:0E:1E:33:D7:1D:6F:6A:34:CC:00:2E:32:FC:34:F8:9D:03:CD:3A:7B X509v3 Subject Alternative Name: DNS:prometheus-k8s.openshift-monitoring.svc, DNS:prometheus-k8s.openshift-monitoring.svc.cluster.local 1.3.6.1.4.1.2312.17.100.2.1: .$83bba262-c991-4827-b53e-2fbe17d6236f Signature Algorithm: sha256WithRSAEncryption 3b:5d:61:f5:7b:e9:53:62:84:e2:51:dd:05:1a:56:81:35:92: a7:fb:ec:9a:ad:18:f5:be:15:e1:08:6a:80:4d:13:64:df:cd: ce:f7:e9:08:f4:12:72:35:4b:e7:89:61:2e:ef:62:b1:80:c5: 7e:0d:94:5f:16:93:b3:9f:58:8c:cb:f1:c7:71:b6:d1:69:3c: 6e:59:1a:2d:6c:44:f4:8f:13:a7:cc:47:f2:08:3d:61:48:c7: 0e:09:47:73:8e:64:02:85:dc:b4:9d:83:37:30:7c:d1:36:b0: 8c:a7:20:0b:70:f2:4a:32:56:72:c5:1d:85:8e:a9:2e:91:65: c6:bb:d0:e3:d6:3d:77:b2:66:95:67:95:15:15:cb:5c:28:a2: df:cf:91:3f:10:91:b3:83:95:1f:ef:d5:c2:68:7e:5e:f1:87: c3:a4:b7:17:b6:3b:ed:74:7a:60:94:5f:50:93:b7:29:75:93: 8c:01:c8:bb:b9:26:d4:76:1e:63:8c:9d:46:10:5e:53:d5:e2: 0b:9d:a3:ea:bf:d3:37:9f:c0:2a:f6:36:d5:b6:bc:1e:0e:bf: aa:4a:ca:63:5a:19:a9:19:5a:9d:43:aa:06:76:70:b5:ba:9e: 0b:71:a4:70:57:76:e2:bd:fe:ed:4f:79:9b:61:5e:fa:d4:15: d5:f0:c0:e5 Certificate: Data: Version: 3 (0x2) Serial Number: 559307143288823574 (0x7c30ee22c119f16) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=openshift-service-serving-signer@1587519388 Validity Not Before: Apr 22 01:36:27 2020 GMT Not After : Jun 21 01:36:28 2022 GMT Subject: CN=openshift-service-serving-signer@1587519388 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:d7:6c:71:3d:08:d4:bd:f1:02:20:02:64:dc:07: 6d:98:49:d1:63:cb:9e:c0:0b:2e:c1:b5:d2:1d:0e: 58:39:2a:35:f2:0f:26:05:61:ca:c5:49:97:2f:cb: 9f:2f:55:21:58:02:2e:3e:40:34:6f:cb:06:92:72: 83:eb:b7:8a:1c:51:3f:c6:b7:18:9f:7b:95:6c:c4: 99:b6:80:df:5c:1c:2e:75:45:a9:8a:08:a6:fa:e8: 56:c1:36:38:7f:f2:2f:f8:01:d9:34:9f:46:78:6c: 6b:62:0b:2b:0e:8e:36:e2:6c:65:4a:a3:dd:1d:2c: e5:2c:2d:2d:d3:c2:01:4c:21:08:18:84:fd:b7:36: 2d:67:15:75:47:e7:5c:27:64:31:a2:e7:25:5f:15: 98:19:a2:32:73:2e:7b:e3:60:aa:2f:38:0b:6d:1d: 1e:82:f2:40:90:36:09:86:57:17:9e:e5:63:1f:46: 2f:7f:3c:3f:3d:fc:43:ba:91:22:c9:c3:ad:5d:ff: b4:09:a2:06:0d:c5:82:ad:d9:d0:84:3e:fe:4b:7d: db:da:9e:aa:04:30:44:9e:d7:e8:89:6f:1c:4e:09: 7c:d7:af:93:ca:43:60:0c:e9:bb:b8:10:21:9e:d0: 56:bb:4c:0f:a3:4e:f9:73:0d:1d:c7:1c:a5:0f:0d: 94:79 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: 0E:1E:33:D7:1D:6F:6A:34:CC:00:2E:32:FC:34:F8:9D:03:CD:3A:7B X509v3 Authority Key Identifier: keyid:0E:1E:33:D7:1D:6F:6A:34:CC:00:2E:32:FC:34:F8:9D:03:CD:3A:7B Signature Algorithm: sha256WithRSAEncryption 30:34:33:47:05:c1:cb:ef:4c:39:66:9f:55:a5:c7:a0:e1:18: 6e:9d:db:5e:07:c1:a4:c8:00:98:12:f8:a2:be:8a:92:2d:1d: a9:64:ad:24:a5:1a:29:be:da:cd:40:37:c3:32:dd:7d:a7:3c: cc:98:7a:40:31:f5:8f:d8:5a:52:a9:42:65:ef:33:f1:18:af: 5e:e9:70:06:33:58:80:16:7f:6b:4c:92:55:9d:f1:84:e5:0d: bb:a6:a4:0f:45:c2:6c:b3:f3:fa:aa:5e:3e:33:d8:1e:13:2a: ea:ef:3b:1e:a0:02:47:28:bf:d3:07:61:0f:aa:4b:56:fc:7a: 44:6b:80:eb:b0:1c:10:fb:7b:48:ca:b4:5b:3a:f5:20:94:03: 7a:cb:5e:34:92:cf:f2:f8:ac:e8:64:83:88:b3:2a:ec:ac:ba: 4b:06:a1:2c:9f:be:ba:dd:d1:6f:78:1a:a8:60:cb:39:64:45: fd:2b:9c:fb:04:b5:42:08:61:0b:3d:23:da:6a:31:dc:1b:14: e0:f9:4c:27:04:22:f7:2f:0b:c3:19:e8:9f:d1:00:ef:18:15: f1:30:5e:9e:fb:60:6f:a9:f7:97:a5:50:a1:4e:87:38:ad:ef: 14:5d:f4:fb:7a:5c:8e:60:3f:f0:27:7f:c1:ed:94:3e:f3:4e: 69:4e:7e:ad Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-04-21-233210 How reproducible: always Steps to Reproduce: 1. Login monitoring routes 2. 3. Actual results: Authentication "500 Internal Error" for all monitoring routes Expected results: no error Additional info:
I believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1826259. Can you check with a release that contains the fix and possibly close as duplicate if my hunch proves to be true?
*** This bug has been marked as a duplicate of bug 1826259 ***
Tested with the fix include bug 1826259, all monitoring routes could be accessed