Bug 1826601

Summary:

Authentication "500 Internal Error" for all monitoring routes

Product:

OpenShift Container Platform

Reporter:

Junqi Zhao <juzhao>

Component:

oauth-proxy

Assignee:

Standa Laznicka <slaznick>

Status:

CLOSED DUPLICATE

QA Contact:

Junqi Zhao <juzhao>

Severity:

high

Docs Contact:

Priority:

high

Version:

4.5

CC:

aos-bugs, mfojtik, slaznick

Target Milestone:

---

Keywords:

Regression

Target Release:

4.5.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-04-24 03:55:48 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Authentication "500 Internal Error"	none

Description Junqi Zhao 2020-04-22 04:15:16 UTC

Created attachment 1680730 [details]
Authentication "500 Internal Error"

Created attachment 1680730 [details]
Authentication "500 Internal Error"

Description of problem:
Authentication "500 Internal Error" for all monitoring routes
# kubectl -n openshift-monitoring get route
NAME                HOST/PORT                                                                           PATH   SERVICES            PORT    TERMINATION          WILDCARD
alertmanager-main   alertmanager-main-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com          alertmanager-main   web     reencrypt/Redirect   None
grafana             grafana-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com                    grafana             https   reencrypt/Redirect   None
prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com             prometheus-k8s      web     reencrypt/Redirect   None
thanos-querier      thanos-querier-openshift-monitoring.apps.juzhao-45.qe.devcluster.openshift.com             thanos-querier      web     reencrypt/Redirect   None

take prometheus route as an example, the UI shows 500 Internal Error, also see the error in container logs
# oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus-proxy
2020/04/22 01:49:43 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s
2020/04/22 01:49:43 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2020/04/22 01:49:43 provider.go:312: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2020/04/22 01:49:43 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:9090/"
2020/04/22 01:49:43 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"
2020/04/22 01:49:43 oauthproxy.go:227: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s
2020/04/22 01:49:43 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled
2020/04/22 01:49:43 main.go:154: using htpasswd file /etc/proxy/htpasswd/auth
2020/04/22 01:49:43 http.go:107: HTTPS: listening on [::]:9091
I0422 01:49:43.260202       1 dynamic_serving_content.go:129] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
2020/04/22 02:55:11 provider.go:394: authorizer reason: 
2020/04/22 02:55:12 provider.go:394: authorizer reason: 
2020/04/22 02:55:13 provider.go:575: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2020/04/22 02:55:13 provider.go:615: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2020/04/22 02:55:18 provider.go:575: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2020/04/22 02:55:18 provider.go:615: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
  "issuer": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com",
  "authorization_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/authorize",
  "token_endpoint": "https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/token",
  "scopes_supported": [
    "user:check-access",
    "user:full",
    "user:info",
    "user:list-projects",
    "user:list-scoped-projects"
  ],
  "response_types_supported": [
    "code",
    "token"
  ],
  "grant_types_supported": [
    "authorization_code",
    "implicit"
  ],
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ]
}
2020/04/22 02:55:18 oauthproxy.go:645: error redeeming code (client:10.128.2.3:55456): Post https://oauth-openshift.apps.juzhao-45.qe.devcluster.openshift.com/oauth/token: x509: certificate signed by unknown authority
2020/04/22 02:55:18 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error

# openssl crl2pkcs7 -nocrl -certfile <(kubectl -n openshift-monitoring get secret prometheus-k8s-tls -o jsonpath='{.data.tls\.crt}' | base64 -d) | openssl pkcs7 -print_certs -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 5679961306105448820 (0x4ed346b6d4610974)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=openshift-service-serving-signer@1587519388
        Validity
            Not Before: Apr 22 01:48:08 2020 GMT
            Not After : Apr 22 01:48:09 2022 GMT
        Subject: CN=prometheus-k8s.openshift-monitoring.svc
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d3:4e:1a:ca:63:6b:f2:ad:1f:e0:17:f3:d1:21:
                    66:e8:37:d2:b1:d7:7a:ee:91:36:f2:ea:04:44:45:
                    23:91:a6:29:42:aa:b3:90:55:33:22:dc:72:3b:56:
                    9e:33:5d:13:fb:6c:5c:68:bd:76:1f:06:bf:35:f2:
                    33:e6:7d:67:d5:58:89:32:cf:ee:7b:af:03:fa:f1:
                    0b:a6:ff:70:24:c5:83:ed:ab:de:f7:b1:ee:44:b9:
                    8e:a8:ba:6a:d2:54:17:ba:5c:ba:64:8c:d6:a5:a5:
                    b9:86:f9:f1:36:d8:c5:83:36:57:10:5d:b3:65:8a:
                    9b:99:57:7d:28:1c:92:3e:c5:74:66:ab:15:09:4d:
                    e9:78:5a:d4:a1:0f:54:85:a8:3e:45:e6:d3:02:83:
                    e5:31:96:17:4e:7c:61:e0:88:32:c7:25:05:2c:6f:
                    ed:37:2b:8c:e1:68:cf:ad:05:7a:30:f2:d8:9c:ae:
                    3d:63:54:88:29:a1:25:d7:3a:1b:75:fd:b8:ce:c3:
                    47:85:8b:cf:b0:9f:fc:e8:ad:66:1e:b5:56:3c:a5:
                    bb:27:07:fd:25:d3:27:c9:04:af:7a:29:1d:48:c1:
                    db:21:ba:83:6b:2b:cb:01:fa:85:b5:08:05:6e:40:
                    ac:94:f3:27:7f:64:42:6b:90:73:d4:b8:61:fb:50:
                    6e:6f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                5B:11:44:7A:AC:BE:F0:6C:E7:B6:9F:60:80:FE:D2:C0:4E:4C:33:9E
            X509v3 Authority Key Identifier: 
                keyid:0E:1E:33:D7:1D:6F:6A:34:CC:00:2E:32:FC:34:F8:9D:03:CD:3A:7B

            X509v3 Subject Alternative Name: 
                DNS:prometheus-k8s.openshift-monitoring.svc, DNS:prometheus-k8s.openshift-monitoring.svc.cluster.local
            1.3.6.1.4.1.2312.17.100.2.1: 
                .$83bba262-c991-4827-b53e-2fbe17d6236f
    Signature Algorithm: sha256WithRSAEncryption
         3b:5d:61:f5:7b:e9:53:62:84:e2:51:dd:05:1a:56:81:35:92:
         a7:fb:ec:9a:ad:18:f5:be:15:e1:08:6a:80:4d:13:64:df:cd:
         ce:f7:e9:08:f4:12:72:35:4b:e7:89:61:2e:ef:62:b1:80:c5:
         7e:0d:94:5f:16:93:b3:9f:58:8c:cb:f1:c7:71:b6:d1:69:3c:
         6e:59:1a:2d:6c:44:f4:8f:13:a7:cc:47:f2:08:3d:61:48:c7:
         0e:09:47:73:8e:64:02:85:dc:b4:9d:83:37:30:7c:d1:36:b0:
         8c:a7:20:0b:70:f2:4a:32:56:72:c5:1d:85:8e:a9:2e:91:65:
         c6:bb:d0:e3:d6:3d:77:b2:66:95:67:95:15:15:cb:5c:28:a2:
         df:cf:91:3f:10:91:b3:83:95:1f:ef:d5:c2:68:7e:5e:f1:87:
         c3:a4:b7:17:b6:3b:ed:74:7a:60:94:5f:50:93:b7:29:75:93:
         8c:01:c8:bb:b9:26:d4:76:1e:63:8c:9d:46:10:5e:53:d5:e2:
         0b:9d:a3:ea:bf:d3:37:9f:c0:2a:f6:36:d5:b6:bc:1e:0e:bf:
         aa:4a:ca:63:5a:19:a9:19:5a:9d:43:aa:06:76:70:b5:ba:9e:
         0b:71:a4:70:57:76:e2:bd:fe:ed:4f:79:9b:61:5e:fa:d4:15:
         d5:f0:c0:e5

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 559307143288823574 (0x7c30ee22c119f16)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=openshift-service-serving-signer@1587519388
        Validity
            Not Before: Apr 22 01:36:27 2020 GMT
            Not After : Jun 21 01:36:28 2022 GMT
        Subject: CN=openshift-service-serving-signer@1587519388
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d7:6c:71:3d:08:d4:bd:f1:02:20:02:64:dc:07:
                    6d:98:49:d1:63:cb:9e:c0:0b:2e:c1:b5:d2:1d:0e:
                    58:39:2a:35:f2:0f:26:05:61:ca:c5:49:97:2f:cb:
                    9f:2f:55:21:58:02:2e:3e:40:34:6f:cb:06:92:72:
                    83:eb:b7:8a:1c:51:3f:c6:b7:18:9f:7b:95:6c:c4:
                    99:b6:80:df:5c:1c:2e:75:45:a9:8a:08:a6:fa:e8:
                    56:c1:36:38:7f:f2:2f:f8:01:d9:34:9f:46:78:6c:
                    6b:62:0b:2b:0e:8e:36:e2:6c:65:4a:a3:dd:1d:2c:
                    e5:2c:2d:2d:d3:c2:01:4c:21:08:18:84:fd:b7:36:
                    2d:67:15:75:47:e7:5c:27:64:31:a2:e7:25:5f:15:
                    98:19:a2:32:73:2e:7b:e3:60:aa:2f:38:0b:6d:1d:
                    1e:82:f2:40:90:36:09:86:57:17:9e:e5:63:1f:46:
                    2f:7f:3c:3f:3d:fc:43:ba:91:22:c9:c3:ad:5d:ff:
                    b4:09:a2:06:0d:c5:82:ad:d9:d0:84:3e:fe:4b:7d:
                    db:da:9e:aa:04:30:44:9e:d7:e8:89:6f:1c:4e:09:
                    7c:d7:af:93:ca:43:60:0c:e9:bb:b8:10:21:9e:d0:
                    56:bb:4c:0f:a3:4e:f9:73:0d:1d:c7:1c:a5:0f:0d:
                    94:79
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                0E:1E:33:D7:1D:6F:6A:34:CC:00:2E:32:FC:34:F8:9D:03:CD:3A:7B
            X509v3 Authority Key Identifier: 
                keyid:0E:1E:33:D7:1D:6F:6A:34:CC:00:2E:32:FC:34:F8:9D:03:CD:3A:7B

    Signature Algorithm: sha256WithRSAEncryption
         30:34:33:47:05:c1:cb:ef:4c:39:66:9f:55:a5:c7:a0:e1:18:
         6e:9d:db:5e:07:c1:a4:c8:00:98:12:f8:a2:be:8a:92:2d:1d:
         a9:64:ad:24:a5:1a:29:be:da:cd:40:37:c3:32:dd:7d:a7:3c:
         cc:98:7a:40:31:f5:8f:d8:5a:52:a9:42:65:ef:33:f1:18:af:
         5e:e9:70:06:33:58:80:16:7f:6b:4c:92:55:9d:f1:84:e5:0d:
         bb:a6:a4:0f:45:c2:6c:b3:f3:fa:aa:5e:3e:33:d8:1e:13:2a:
         ea:ef:3b:1e:a0:02:47:28:bf:d3:07:61:0f:aa:4b:56:fc:7a:
         44:6b:80:eb:b0:1c:10:fb:7b:48:ca:b4:5b:3a:f5:20:94:03:
         7a:cb:5e:34:92:cf:f2:f8:ac:e8:64:83:88:b3:2a:ec:ac:ba:
         4b:06:a1:2c:9f:be:ba:dd:d1:6f:78:1a:a8:60:cb:39:64:45:
         fd:2b:9c:fb:04:b5:42:08:61:0b:3d:23:da:6a:31:dc:1b:14:
         e0:f9:4c:27:04:22:f7:2f:0b:c3:19:e8:9f:d1:00:ef:18:15:
         f1:30:5e:9e:fb:60:6f:a9:f7:97:a5:50:a1:4e:87:38:ad:ef:
         14:5d:f4:fb:7a:5c:8e:60:3f:f0:27:7f:c1:ed:94:3e:f3:4e:
         69:4e:7e:ad

Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-04-21-233210

How reproducible:
always

Steps to Reproduce:
1. Login monitoring routes
2.
3.

Actual results:
Authentication "500 Internal Error" for all monitoring routes

Expected results:
no error

Additional info:

Comment 1 Standa Laznicka 2020-04-22 11:52:07 UTC

I believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1826259. Can you check with a release that contains the fix and possibly close as duplicate if my hunch proves to be true?

Comment 3 Junqi Zhao 2020-04-24 03:55:48 UTC


*** This bug has been marked as a duplicate of bug 1826259 ***

Comment 4 Junqi Zhao 2020-04-24 06:37:10 UTC

Tested with the fix include bug 1826259, all monitoring routes could be accessed