Bug 1867494

Summary: webhook authenticator does not have the master name replaced
Product: OpenShift Container Platform Reporter: RamaKasturi <knarra>
Component: kube-controller-managerAssignee: Tomáš Nožička <tnozicka>
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.6CC: aos-bugs, mfojtik, tnozicka
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:26:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description RamaKasturi 2020-08-10 07:39:50 UTC 
When verifying bug https://bugzilla.redhat.com/show_bug.cgi?id=1842002 hit error below when kube-apiserver is down.

E0807 13:48:12.608278       1 webhook.go:111] Failed to make webhook authenticator request: Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused
E0807 13:48:12.608328       1 authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused]

Version-Release number of selected component (if applicable):
Client Version: 4.6.0-202008031851.p0-a695d74
Server Version: 4.6.0-0.nightly-2020-08-07-042746

How Reproducible:
Always

Steps to Reproduce:
1) Install 4.6 cluster
2) on the same master, bring down the local kube-apiserver pod and see if the local KCM fails or keeps working use a master with KCM leader
3) To bring down local kube-apiserver, ssh to the master , mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /home/kube-apiserver-pod.yaml
4) Terminate it gracefully by running the command "pid=$(ps aux | grep " kube-apiserver " | grep -v grep  | awk 'NR==1 {print $2}');
kill $pid"
5) Now keep checking KCM and api-server logs

Actual Results:
=================
Below errors appear in the KCM logs:
E0807 13:48:12.608278       1 webhook.go:111] Failed to make webhook authenticator request: Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused
E0807 13:48:12.608328       1 authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused]

Expected Results:
======================
Should not see any errors as above.

Additional Info:
https://coreos.slack.com/archives/CKJR6200N/p1596805775023000 - More info in this thread
Comment 1 RamaKasturi 2020-08-10 07:40:47 UTC 
Raising this bug in kcm as i am not sure which component to assign  for , tomas could you please help reassign, thanks !!
Comment 2 Tomáš Nožička 2020-08-10 08:12:47 UTC 
kcm is the correct choice ;)
Comment 6 RamaKasturi 2020-09-11 06:50:24 UTC 
Verified bug with the payload below and i do not see any messages as reported. So marking the bug verified.

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-10-195619   True        False         49m     Cluster version is 4.6.0-0.nightly-2020-09-10-195619

Followed the same steps as the steps used to reproduce the issue.
Comment 9 errata-xmlrpc 2020-10-27 16:26:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196