Bug 1867494 - webhook authenticator does not have the master name replaced
Summary: webhook authenticator does not have the master name replaced
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Tomáš Nožička
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-10 07:39 UTC by RamaKasturi
Modified: 2020-09-11 07:22 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-kube-controller-manager-operator pull 441 None closed Bug 1867494: Explicitly use internal LB for KCM and CPC 2020-09-21 06:18:39 UTC

Description RamaKasturi 2020-08-10 07:39:50 UTC
When verifying bug https://bugzilla.redhat.com/show_bug.cgi?id=1842002 hit error below when kube-apiserver is down.

E0807 13:48:12.608278       1 webhook.go:111] Failed to make webhook authenticator request: Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused
E0807 13:48:12.608328       1 authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused]

Version-Release number of selected component (if applicable):
Client Version: 4.6.0-202008031851.p0-a695d74
Server Version: 4.6.0-0.nightly-2020-08-07-042746

How Reproducible:
Always

Steps to Reproduce:
1) Install 4.6 cluster
2) on the same master, bring down the local kube-apiserver pod and see if the local KCM fails or keeps working use a master with KCM leader
3) To bring down local kube-apiserver, ssh to the master , mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /home/kube-apiserver-pod.yaml
4) Terminate it gracefully by running the command "pid=$(ps aux | grep " kube-apiserver " | grep -v grep  | awk 'NR==1 {print $2}');
kill $pid"
5) Now keep checking KCM and api-server logs

Actual Results:
=================
Below errors appear in the KCM logs:
E0807 13:48:12.608278       1 webhook.go:111] Failed to make webhook authenticator request: Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused
E0807 13:48:12.608328       1 authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, Post "https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp [::1]:6443: connect: connection refused]

Expected Results:
======================
Should not see any errors as above.

Additional Info:
https://coreos.slack.com/archives/CKJR6200N/p1596805775023000 - More info in this thread

Comment 1 RamaKasturi 2020-08-10 07:40:47 UTC
Raising this bug in kcm as i am not sure which component to assign  for , tomas could you please help reassign, thanks !!

Comment 2 Tomáš Nožička 2020-08-10 08:12:47 UTC
kcm is the correct choice ;)

Comment 6 RamaKasturi 2020-09-11 06:50:24 UTC
Verified bug with the payload below and i do not see any messages as reported. So marking the bug verified.

[ramakasturinarra@dhcp35-60 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-10-195619   True        False         49m     Cluster version is 4.6.0-0.nightly-2020-09-10-195619

Followed the same steps as the steps used to reproduce the issue.


Note You need to log in before you can comment on or make changes to this bug.