Bug 1531938

Summary:	Log Message Spam if ClusterRoleBinding exists for non-existent ClusterRole
Product:	OpenShift Container Platform	Reporter:	Stefanie Forrester <dakini>
Component:	apiserver-auth	Assignee:	Mo <mkhan>
Status:	CLOSED ERRATA	QA Contact:	Chuan Yu <chuyu>
Severity:	low	Docs Contact:
Priority:	low
Version:	3.7.1	CC:	aos-bugs, eparis, evb, fshaikh, jialiu, jokerman, jolee, knewcomer, ksalunkh, mkhan, mmccomas, nagrawal, nmoraiti, pkanthal, rbost, rgregory, ssorce, tparsons, xtian
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 10:40:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Stefanie Forrester 2018-01-06 18:51:09 UTC

Description of problem:

The journal is being spammed with this message about 850 times per second:

Jan 06 18:42:09 ip-172-31-54-162.ec2.internal atomic-openshift-master-api[43301]: E0106 18:42:09.730919   43301 cache.go:332] error synchronizing: clusterrole.rbac.authorization.k8s.io "cluster-capacity-role" not found

[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="3 minutes ago" |grep '18:42:09' |grep -c clusterrole.rbac.authorization.k8s.io 
850
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="3 minutes ago" |grep '18:42:10' |grep -c clusterrole.rbac.authorization.k8s.io 
843
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="5 minutes ago" |grep '18:42:11' |grep -c clusterrole.rbac.authorization.k8s.io 
850


Version-Release number of selected component (if applicable):

oc v3.7.9

How reproducible:

It's happening on all 3 masters in starter-us-east-1. I haven't seen it elsewhere.

Steps to Reproduce:
1. journalctl -fl
2.
3.

Actual results:

Many instances of this message scroll by every second:

Jan 06 18:48:16 ip-172-31-55-199.ec2.internal atomic-openshift-master-api[16734]: E0106 18:48:16.652886   16734 cache.go:332] error synchronizing: clusterrole.rbac.authorization.k8s.io "cluster-capacity-role" not found


Expected results:


Additional info:

Comment 1 Eric Paris 2018-01-09 14:48:32 UTC

There was a role binding, but no role. I have deleted the invalid role binding from the cluster in question to work around this BZ.

We should log, maybe even regularly. But not 850 times per second.

Comment 2 Simo Sorce 2018-01-19 15:46:36 UTC

Eric I am sympatethic to the request, but how do you propose we handle this?
Should we write a new logging library that can deal with this ?

Comment 3 Eric Paris 2018-02-08 17:53:54 UTC

If the message needs to be logged, rate limit it yourself (write a helper so others can use it if it doesn't exist).  If we don't really need to know about this message and/or if there is nothing the admin can or should do to correct the situtation, we should reduce the log level.

Comment 5 Mo 2018-03-05 14:18:08 UTC

WIP upstream PR https://github.com/kubernetes/kubernetes/pull/58307

Comment 13 kedar 2018-12-27 05:46:43 UTC

Hello,

Do we have any update on this issue.

Thanks,
Kedar Salunkhe

Comment 14 Neelesh Agrawal 2019-03-06 17:05:54 UTC

This has been fixed with https://github.com/openshift/origin/pull/21522

Comment 15 Chuan Yu 2019-03-08 06:18:52 UTC

Verified.

I have check the kube api log, no such log report now.

oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-03-06-074438   True        False         43m       Cluster version is 4.0.0-0.nightly-2019-03-06-074438

Comment 18 Mo 2019-05-09 19:18:47 UTC

*** Bug 1699937 has been marked as a duplicate of this bug. ***

Comment 20 errata-xmlrpc 2019-06-04 10:40:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758

Comment 21 Red Hat Bugzilla 2023-09-15 00:06:01 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days