Bug 1907380 - kube-rbac-proxy exposes tokens, has excessive verbosity
Summary: kube-rbac-proxy exposes tokens, has excessive verbosity
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.7.0
Assignee: Joel Speed
QA Contact: Milind Yadav
URL:
Whiteboard:
Depends On:
Blocks: 1908316
TreeView+ depends on / blocked
 
Reported: 2020-12-14 11:42 UTC by Vadim Rutkovsky
Modified: 2021-02-24 15:43 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The log level for the kube-rbac-proxy was set to level 10 Consequence: Logs messages contained excessive detail about the operations of the kube-rbac-proxy. Much more than needed in most scenarios Fix: Reduce the log level to 3 to match other components Result: Log messages have been reduced and are much more appropriate
Clone Of:
Environment:
Last Closed: 2021-02-24 15:43:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-autoscaler-operator pull 189 0 None closed Bug 1907380: Reduce verbosity of kube-rbac-proxy logging 2021-02-18 02:22:24 UTC
Github openshift cluster-machine-approver pull 98 0 None closed Bug 1907380: Reduce verbosity of kube-rbac-proxy logging 2021-02-18 02:22:24 UTC
Github openshift machine-api-operator pull 776 0 None closed Bug 1907380: Reduce verbosity of kube-rbac-proxy logging 2021-02-18 02:22:24 UTC
Github openshift machine-api-operator pull 781 0 None closed Bug 1907380: Reduce verbosity of kube-rbac-proxy logging for operator 2021-02-18 02:22:25 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:43:59 UTC

Description Vadim Rutkovsky 2020-12-14 11:42:03 UTC
Description of problem:
kube-rbac-proxy container in various components of `openshift-machine-api` has excessive verbosity:
* unnecessarily logs too many lines which decreases disk performance
* logs bearer tokens, which is insecure

This seems to happen for all providers - noticed in AWS and vSphere tests at least

Comment 1 Vadim Rutkovsky 2020-12-14 11:42:57 UTC
Same for pods in `openshift-cluster-machine-approver` namespace

Comment 3 Milind Yadav 2020-12-18 05:04:10 UTC
Validaed at : 
[miyadav@miyadav aws]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-12-17-201522   True        False         47m     Cluster version is 4.7.0-0.nightly-2020-12-17-201522



Steps :  I scaled machineset , used mhc to manage a deleted node , all time below is the only log I got .


[miyadav@miyadav aws]$ oc logs -f machine-api-controllers-668dfc9bd4-92sh5 -c kube-rbac-proxy-machine-mtrc
I1218 04:13:49.935863       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I1218 04:13:49.938152       1 main.go:190] Valid token audiences: 
I1218 04:13:49.938320       1 main.go:278] Reading certificate files
I1218 04:13:49.938620       1 main.go:311] Starting TCP socket on 0.0.0.0:8441
I1218 04:13:49.939084       1 main.go:318] Listening securely on 0.0.0.0:8441
^C
[miyadav@miyadav aws]$ oc logs -f machine-api-controllers-668dfc9bd4-92sh5 -c kube-rbac-proxy-machineset-mtrc
I1218 04:13:49.679787       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I1218 04:13:49.681627       1 main.go:190] Valid token audiences: 
I1218 04:13:49.681698       1 main.go:278] Reading certificate files
I1218 04:13:49.681971       1 main.go:311] Starting TCP socket on 0.0.0.0:8442
I1218 04:13:49.682383       1 main.go:318] Listening securely on 0.0.0.0:8442
^C
[miyadav@miyadav aws]$ oc logs -f machine-api-controllers-668dfc9bd4-92sh5 -c kube-rbac-proxy-mhc-mtrc
I1218 04:13:50.220234       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I1218 04:13:50.221931       1 main.go:190] Valid token audiences: 
I1218 04:13:50.221981       1 main.go:278] Reading certificate files
I1218 04:13:50.222200       1 main.go:311] Starting TCP socket on 0.0.0.0:8444
I1218 04:13:50.222879       1 main.go:318] Listening securely on 0.0.0.0:8444

for pods in `openshift-cluster-machine-approver` namespace


[miyadav@miyadav aws]$ oc logs -f machine-approver-7f68d779cc-2448t -c kube-rbac-proxy 
I1218 04:13:16.733138       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I1218 04:13:16.784447       1 main.go:190] Valid token audiences: 
I1218 04:13:16.784663       1 main.go:278] Reading certificate files
I1218 04:13:16.785097       1 main.go:311] Starting TCP socket on 0.0.0.0:9192
I1218 04:13:16.785587       1 main.go:318] Listening securely on 0.0.0.0:9192
^C
[miyadav@miyadav aws]$ oc logs  machine-approver-7f68d779cc-2448t -c kube-rbac-proxy 
I1218 04:13:16.733138       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I1218 04:13:16.784447       1 main.go:190] Valid token audiences: 
I1218 04:13:16.784663       1 main.go:278] Reading certificate files
I1218 04:13:16.785097       1 main.go:311] Starting TCP socket on 0.0.0.0:9192
I1218 04:13:16.785587       1 main.go:318] Listening securely on 0.0.0.0:9192

Additional info :

Moved to VERIFIED

Comment 4 Milind Yadav 2020-12-18 08:08:30 UTC
the loglevel still is 10 for kube-rbac containers inside pods -- machine-api-operator  and cluster-autoscaler-operator hence moving to assigned again

Comment 6 Milind Yadav 2021-01-11 08:17:10 UTC
[miyadav@miyadav Downloads]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-01-10-070949   True        False         116m    Cluster version is 4.7.0-0.nightly-2021-01-10-070949

Steps :

[miyadav@miyadav Downloads]$ oc logs  cluster-autoscaler-operator-c48656855-j2tqv -c kube-rbac-proxy
I0111 05:53:17.294506       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I0111 05:53:17.295693       1 main.go:190] Valid token audiences: 
I0111 05:53:17.295731       1 main.go:278] Reading certificate files
I0111 05:53:17.295909       1 main.go:311] Starting TCP socket on 0.0.0.0:9192
I0111 05:53:17.296248       1 main.go:318] Listening securely on 0.0.0.0:9192

[miyadav@miyadav Downloads]$ oc logs  machine-api-controllers-6695566-8tc5j 
error: a container name must be specified for pod machine-api-controllers-6695566-8tc5j, choose one of: [machineset-controller machine-controller nodelink-controller machine-healthcheck-controller kube-rbac-proxy-machineset-mtrc kube-rbac-proxy-machine-mtrc kube-rbac-proxy-mhc-mtrc]
[miyadav@miyadav Downloads]$ oc logs  machine-api-controllers-6695566-8tc5j -c kube-rbac-proxy-mhc-mtrc
I0111 05:53:24.991551       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I0111 05:53:24.992746       1 main.go:190] Valid token audiences: 
I0111 05:53:24.992782       1 main.go:278] Reading certificate files
I0111 05:53:24.992957       1 main.go:311] Starting TCP socket on 0.0.0.0:8444
I0111 05:53:24.993262       1 main.go:318] Listening securely on 0.0.0.0:8444

[miyadav@miyadav Downloads]$ oc logs  machine-api-operator-78b9ddc44d-79nz7 -c kube-rbac-proxy 
I0111 05:52:05.844006       1 main.go:159] Reading config file: /etc/kube-rbac-proxy/config-file.yaml
I0111 05:52:05.845689       1 main.go:190] Valid token audiences: 
I0111 05:52:05.845739       1 main.go:278] Reading certificate files
I0111 05:52:05.845991       1 main.go:311] Starting TCP socket on 0.0.0.0:8443
I0111 05:52:05.846425       1 main.go:318] Listening securely on 0.0.0.0:8443
.
.
.

Additional Info:
Moved to VERIFIED

Comment 9 errata-xmlrpc 2021-02-24 15:43:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.