Bug 1658899
| Summary: | Continuous error "TLS handshake error" in grafana-proxy container logs | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||
| Component: | Monitoring | Assignee: | Simon Pasquier <spasquie> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Junqi Zhao <juzhao> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.1.z | CC: | crarobin, minden, scuppett, slaznick, surbania, wking | ||||
| Target Milestone: | --- | Keywords: | Reopened | ||||
| Target Release: | 4.4.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Cause: the readiness probe of the OAuth proxy container was misconfigured.
Consequence: the container's logs were flooded by error messages every 10 seconds.
Fix: the readiness probe was configured with the proper settings.
Result: there are no error in the logs anymore.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-05-04 11:12:48 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Junqi Zhao
2018-12-13 06:36:15 UTC
I suspect this will be fixed by https://github.com/openshift/installer/pull/924 I can confirm that https://github.com/openshift/installer/pull/924 does not fix this. I have installed a cluster with that change made (I used v0.12.0), and the error is still occurring despite the ${var.cluster_name}-api-int LB group having these healthcheck params: Protocol: HTTPS Path: /healthz Port: 6443 Healthy threshold: 3 Unhealthy threshold: 3 Timeout: 10 Interval: 10 Success codes: 200-399 not fixed, still see the error # oc -n openshift-monitoring logs grafana-78765ddcc7-7n8zz -c grafana-proxy .................................................... 2019/02/21 03:20:41 server.go:2923: http: TLS handshake error from 10.128.2.1:60838: EOF 2019/02/21 03:20:51 server.go:2923: http: TLS handshake error from 10.128.2.1:60892: EOF 2019/02/21 03:21:01 server.go:2923: http: TLS handshake error from 10.128.2.1:60946: EOF 2019/02/21 03:21:11 server.go:2923: http: TLS handshake error from 10.128.2.1:32874: EOF 2019/02/21 03:21:21 server.go:2923: http: TLS handshake error from 10.128.2.1:32928: EOF 2019/02/21 03:21:31 server.go:2923: http: TLS handshake error from 10.128.2.1:32984: EOF 2019/02/21 03:21:41 server.go:2923: http: TLS handshake error from 10.128.2.1:33036: EOF 2019/02/21 03:21:51 server.go:2923: http: TLS handshake error from 10.128.2.1:33088: EOF 2019/02/21 03:22:01 server.go:2923: http: TLS handshake error from 10.128.2.1:33170: EOF 2019/02/21 03:22:11 server.go:2923: http: TLS handshake error from 10.128.2.1:33224: EOF 2019/02/21 03:22:21 server.go:2923: http: TLS handshake error from 10.128.2.1:33276: EOF 2019/02/21 03:22:31 server.go:2923: http: TLS handshake error from 10.128.2.1:33342: EOF 2019/02/21 03:22:41 server.go:2923: http: TLS handshake error from 10.128.2.1:33542: EOF 2019/02/21 03:22:51 server.go:2923: http: TLS handshake error from 10.128.2.1:33626: EOF 2019/02/21 03:23:01 server.go:2923: http: TLS handshake error from 10.128.2.1:33710: EOF 2019/02/21 03:23:11 server.go:2923: http: TLS handshake error from 10.128.2.1:33784: EOF 2019/02/21 03:23:21 server.go:2923: http: TLS handshake error from 10.128.2.1:33868: EOF 2019/02/21 03:23:31 server.go:2923: http: TLS handshake error from 10.128.2.1:33940: EOF 2019/02/21 03:23:41 server.go:2923: http: TLS handshake error from 10.128.2.1:34010: EOF 2019/02/21 03:23:51 server.go:2923: http: TLS handshake error from 10.128.2.1:34084: EOF # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-02-20-194410 True False 24h Cluster version is 4.0.0-0.nightly-2019-02-20-194410 This was observed in 4.4. It is still worth investigating IMO, why waste customers' storage if looking into the error might unveil more serious issues? Tested with 4.4.0-0.nightly-2020-01-24-141203, issue is fixed # oc -n openshift-monitoring logs grafana-bbb6fcc-qf2j4 -c grafana-proxy 2020/01/26 23:42:28 provider.go:118: Defaulting client-id to system:serviceaccount:openshift-monitoring:grafana 2020/01/26 23:42:28 provider.go:123: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token 2020/01/26 23:42:28 provider.go:311: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates. 2020/01/26 23:42:28 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:3001/" 2020/01/26 23:42:28 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics" 2020/01/26 23:42:28 oauthproxy.go:227: OAuthProxy configured for Client ID: system:serviceaccount:openshift-monitoring:grafana 2020/01/26 23:42:28 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled 2020/01/26 23:42:28 http.go:96: HTTPS: listening on [::]:3000 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |