Bug 1820577

Summary: Fresh Azure cloud cluster's KAS pod logs spam with many "http: TLS handshake error from 168.63.129.16:<port>: EOF"
Product: OpenShift Container Platform Reporter: Xingxing Xia <xxia>
Component: InstallerAssignee: Fabiano Franz <ffranz>
Installer sub component: openshift-installer QA Contact: Johnny Liu <jialiu>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: adahiya, aos-bugs, kewang, mfojtik
Version: 4.4Keywords: Reopened
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-22 23:52:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xingxing Xia 2020-04-03 11:26:29 UTC
Description of problem:
In Azure cloud cluster, EVEN in fresh env without any user operation, KAS pod logs spam with many "http: TLS handshake error from 168.63.129.16:<port>: EOF"

Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-04-02-130551

How reproducible:
Always

Steps to Reproduce:
1. Launch fresh Azure IPI env successfully.
2. Without any user operation, check KAS pod logs
$ oc logs -n openshift-kube-apiserver -c kube-apiserver kube-apiserver-xxia1-0403-tkfs8-master-0 --tail 1 -f

Actual results:
2. EVEN in fresh env given NO any user operation, the logs spam frequently with many below errors:
I0403 10:27:08.835307       1 log.go:172] http: TLS handshake error from 168.63.129.16:55477: EOF
I0403 10:27:18.851322       1 log.go:172] http: TLS handshake error from 168.63.129.16:55548: EOF
I0403 10:27:28.866113       1 log.go:172] http: TLS handshake error from 168.63.129.16:55623: EOF
I0403 10:27:38.866170       1 log.go:172] http: TLS handshake error from 168.63.129.16:55712: EOF
I0403 10:27:48.866219       1 log.go:172] http: TLS handshake error from 168.63.129.16:55811: EOF
I0403 10:27:58.866360       1 log.go:172] http: TLS handshake error from 168.63.129.16:55897: EOF
...

Expected results:
2. Because we announce supporting Azure cloud, we should do this as better as possible, thus above errors in FRESH Azure env should not be seen for customer user experience. Though there is bug 1753443#c3 closed, it suggested others to do something to fix the issue. Here we suggest the cluster itself can do something to fix the issue :)

Additional info:

Comment 1 Stefan Schimanski 2020-04-03 15:50:59 UTC
This is very probably a non-https LB health check set up by the Azure installer. Use the http port for that like gcp does.

Comment 2 Abhinav Dahiya 2020-04-27 17:00:08 UTC

*** This bug has been marked as a duplicate of bug 1828382 ***

Comment 3 Ke Wang 2020-05-21 07:59:13 UTC
This bug was reproduced with OCP 4.5.0-0.nightly-2020-05-20-235311 which installed on Disconnected UPI on Azure with RHCOS & Private Cluster.


$ oc logs -n openshift-kube-apiserver -c kube-apiserver kube-apiserver-pdazqeci-0521-05210324-master-1 --tail 1 -f
I0521 07:56:19.059098       1 log.go:172] http: TLS handshake error from 168.63.129.16:57818: EOF
I0521 07:56:29.074520       1 log.go:172] http: TLS handshake error from 168.63.129.16:58155: EOF
I0521 07:56:39.074489       1 log.go:172] http: TLS handshake error from 168.63.129.16:58527: EOF
...

Comment 5 Abhinav Dahiya 2020-05-22 23:52:23 UTC
> (In reply to Ke Wang from comment #3)
> This bug was reproduced with OCP 4.5.0-0.nightly-2020-05-20-235311 which
> installed on Disconnected UPI on Azure with RHCOS & Private Cluster.

The UPI azure bug is separate 
> https://bugzilla.redhat.com/show_bug.cgi?id=1836016

> 
> 
> $ oc logs -n openshift-kube-apiserver -c kube-apiserver
> kube-apiserver-pdazqeci-0521-05210324-master-1 --tail 1 -f
> I0521 07:56:19.059098       1 log.go:172] http: TLS handshake error from
> 168.63.129.16:57818: EOF
> I0521 07:56:29.074520       1 log.go:172] http: TLS handshake error from
> 168.63.129.16:58155: EOF
> I0521 07:56:39.074489       1 log.go:172] http: TLS handshake error from
> 168.63.129.16:58527: EOF
> ...

so closing this bug.

*** This bug has been marked as a duplicate of bug 1828382 ***