Description of problem: The kube-rbac-proxy-federate container inside the prometheus-user-workload-0 pod deployed part of the openshift-user-workload-monitoring namespace throw TLS handshake error. Version-Release number of selected component (if applicable): openshift 4.10.22 How reproducible: Steps to Reproduce: 1. Deploy openshift 4.10.22 cluster 2. Enable monitoring for user-defined projects 3. check logs in the kube-rbac-proxy-federate container: oc logs prometheus-user-workload-0 -n openshift-user-workload-monitoring -c kube-rbac-proxy-federate Actual results: 022/07/27 17:06:53 http: TLS handshake error from 10.131.0.3:58912: write tcp 10.129.2.15:9092->10.131.0.3:58912: write: connection reset by peer 2022/07/27 17:06:53 http: TLS handshake error from 10.128.2.10:33522: write tcp 10.129.2.15:9092->10.128.2.10:33522: write: connection reset by peer 2022/07/27 17:06:58 http: TLS handshake error from 10.131.0.3:59010: write tcp 10.129.2.15:9092->10.131.0.3:59010: write: connection reset by peer 2022/07/27 17:06:58 http: TLS handshake error from 10.128.2.10:33596: write tcp 10.129.2.15:9092->10.128.2.10:33596: write: connection reset by peer 2022/07/27 17:07:03 http: TLS handshake error from 10.131.0.3:59084: write tcp 10.129.2.15:9092->10.131.0.3:59084: write: connection reset by peer Expected results: No TLS handshake error Additional info: This issue happen even if no application is configured to expose metrics
IIUC these logs are triggered by the router health checks: the router will open a TCP connection to the proxy's port and close right away without proceeding with the TLS handshake. We need to validate the assumption with the auth team that maintains kube-rbac-proxy and investigate whether we have ways to tweak the health checks performed by the router.
https://coreos.slack.com/archives/CCH60A77E/p1646915604267019 describes the same issue
As discussed with the network edge team in https://coreos.slack.com/archives/CCH60A77E/p1659372630615569 and https://coreos.slack.com/archives/CCH60A77E/p1646915604267019 this is an expected behavior due to the health check probes from a Route closing the TCP connection without performing a TLS handshake. But those messages are very frequent and might lead to customer questions, so we'd prefer to avoid then if possible nevertheless. Filtering out those log messages would be done in kube-rbac-proxy side. Per discussion with Auth team in https://coreos.slack.com/archives/CB48XQ4KZ/p1659444053063079 BZ https://bugzilla.redhat.com/show_bug.cgi?id=2030726 might show a similar issue. So I'm reassign to Auth team
So as it seems, this is not a bug. It looks like a feature :) So the haproxy makes health checks by opening tcp connections and then resetting it. We can contribute a "don't report empty connections"-flag to kube-rbac-proxy to get rid of the logging. To reduce the logging, we can reduce the interval of health checks by the routes (haproxy): https://docs.openshift.com/container-platform/4.10/networking/routes/route-configuration.html#nw-route-specific-annotations_route-configuration. Do we need to keep this BZ open to track the progress on kube-rbac-proxy or can we close it?
I will close it as of now. If there is a request to keep this bug open, until: 1. the feature has been added to kube-rbac-proxy and 2. the feature is used within our components, then please reopen the BZ.
Hello, Is it possible to keep this case open? I can see that this BZ is considered more as an RFE based on comment #6 so is there any RFE opened to reflect that ?
I created an issue upstream for that: https://github.com/brancz/kube-rbac-proxy/issues/194
Dear reporter, we greatly appreciate the bug you have reported here. Unfortunately, due to migration to a new issue-tracking system (https://issues.redhat.com/), we cannot continue triaging bugs reported in Bugzilla. Since this bug has been stale for multiple days, we, therefore, decided to close this bug. If you think this is a mistake or this bug has a higher priority or severity as set today, please feel free to reopen this bug and tell us why. We are going to move every re-opened bug to https://issues.redhat.com. Thank you for your patience and understanding.