Bug 1531938 - Log Message Spam if ClusterRoleBinding exists for non-existent ClusterRole [NEEDINFO]
Summary: Log Message Spam if ClusterRoleBinding exists for non-existent ClusterRole
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 3.7.1
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.1.0
Assignee: Mo
QA Contact: Chuan Yu
URL:
Whiteboard:
: 1699937 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-06 18:51 UTC by Stefanie Forrester
Modified: 2019-06-04 10:40 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:40:18 UTC
Target Upstream Version:
ksalunkh: needinfo? (mkhan)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:40:28 UTC
Red Hat Knowledge Base (Solution) 3420361 None None None 2018-04-20 21:42:44 UTC

Description Stefanie Forrester 2018-01-06 18:51:09 UTC
Description of problem:

The journal is being spammed with this message about 850 times per second:

Jan 06 18:42:09 ip-172-31-54-162.ec2.internal atomic-openshift-master-api[43301]: E0106 18:42:09.730919   43301 cache.go:332] error synchronizing: clusterrole.rbac.authorization.k8s.io "cluster-capacity-role" not found

[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="3 minutes ago" |grep '18:42:09' |grep -c clusterrole.rbac.authorization.k8s.io 
850
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="3 minutes ago" |grep '18:42:10' |grep -c clusterrole.rbac.authorization.k8s.io 
843
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="5 minutes ago" |grep '18:42:11' |grep -c clusterrole.rbac.authorization.k8s.io 
850


Version-Release number of selected component (if applicable):

oc v3.7.9

How reproducible:

It's happening on all 3 masters in starter-us-east-1. I haven't seen it elsewhere.

Steps to Reproduce:
1. journalctl -fl
2.
3.

Actual results:

Many instances of this message scroll by every second:

Jan 06 18:48:16 ip-172-31-55-199.ec2.internal atomic-openshift-master-api[16734]: E0106 18:48:16.652886   16734 cache.go:332] error synchronizing: clusterrole.rbac.authorization.k8s.io "cluster-capacity-role" not found


Expected results:


Additional info:

Comment 1 Eric Paris 2018-01-09 14:48:32 UTC
There was a role binding, but no role. I have deleted the invalid role binding from the cluster in question to work around this BZ.

We should log, maybe even regularly. But not 850 times per second.

Comment 2 Simo Sorce 2018-01-19 15:46:36 UTC
Eric I am sympatethic to the request, but how do you propose we handle this?
Should we write a new logging library that can deal with this ?

Comment 3 Eric Paris 2018-02-08 17:53:54 UTC
If the message needs to be logged, rate limit it yourself (write a helper so others can use it if it doesn't exist).  If we don't really need to know about this message and/or if there is nothing the admin can or should do to correct the situtation, we should reduce the log level.

Comment 5 Mo 2018-03-05 14:18:08 UTC
WIP upstream PR https://github.com/kubernetes/kubernetes/pull/58307

Comment 13 kedar 2018-12-27 05:46:43 UTC
Hello,

Do we have any update on this issue.

Thanks,
Kedar Salunkhe

Comment 14 Neelesh Agrawal 2019-03-06 17:05:54 UTC
This has been fixed with https://github.com/openshift/origin/pull/21522

Comment 15 Chuan Yu 2019-03-08 06:18:52 UTC
Verified.

I have check the kube api log, no such log report now.

oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-03-06-074438   True        False         43m       Cluster version is 4.0.0-0.nightly-2019-03-06-074438

Comment 18 Mo 2019-05-09 19:18:47 UTC
*** Bug 1699937 has been marked as a duplicate of this bug. ***

Comment 20 errata-xmlrpc 2019-06-04 10:40:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.