Bug 1697875

Summary:	kube-scheduler/controller-manager doesn't have permissions to create subject access reviews
Product:	OpenShift Container Platform	Reporter:	Sergiusz Urbaniak <surbania>
Component:	Master	Assignee:	Mo <mkhan>
Status:	CLOSED ERRATA	QA Contact:	Chuan Yu <chuyu>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.1.0	CC:	aos-bugs, jokerman, mkhan, mmccomas, sponnaga
Target Milestone:	---
Target Release:	4.1.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 10:47:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sergiusz Urbaniak 2019-04-09 08:41:20 UTC

Using the following cluster version:

NAME      VERSION                        AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.ci-2019-04-09-033744   True        False         81m     Cluster version is 4.0.0-0.ci-2019-04-09-033744

Recently, controller manager as well as scheduler moved to TLS in [1] and [2] for their internal endpoints. This broke monitoring, because the internal endpoints need to be scraped for metrics.

Authn and authz have been enabled in [3] and [4], but apparently the kube scheduler and the kube controller manager need more permissions to execut subject access reviews:

controller manager:

E0409 07:46:18.173664       1 errors.go:77] subjectaccessreviews.authorization.k8s.io is forbidden: User "system:kube-controller-manager" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope

The kube scheduler already fails at the token review:

E0409 07:46:37.988933       1 webhook.go:106] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:kube-scheduler" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope
E0409 07:46:37.988971       1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, tokenreviews.authentication.k8s.io is forbidden: User "system:kube-scheduler" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope]

[1] https://github.com/openshift/cluster-kube-scheduler-operator/pull/88
[2] https://github.com/openshift/cluster-kube-controller-manager-operator/pull/207

[3] https://github.com/openshift/cluster-kube-scheduler-operator/pull/89
[4] https://github.com/openshift/cluster-kube-controller-manager-operator/pull/213

This blocks the following PRs:
- https://github.com/openshift/cluster-monitoring-operator/pull/312
- https://github.com/openshift/cluster-monitoring-operator/pull/308

Comment 5 Chuan Yu 2019-05-07 06:27:46 UTC

No such issue for now, mark as verified.
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-05-06-223020   True        False         4h57m   Cluster version is 4.1.0-0.nightly-2019-05-06-223020

Comment 7 errata-xmlrpc 2019-06-04 10:47:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758