Description of problem:
The journal is being spammed with this message about 850 times per second:
Jan 06 18:42:09 ip-172-31-54-162.ec2.internal atomic-openshift-master-api: E0106 18:42:09.730919 43301 cache.go:332] error synchronizing: clusterrole.rbac.authorization.k8s.io "cluster-capacity-role" not found
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="3 minutes ago" |grep '18:42:09' |grep -c clusterrole.rbac.authorization.k8s.io
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="3 minutes ago" |grep '18:42:10' |grep -c clusterrole.rbac.authorization.k8s.io
[root@starter-us-east-1-master-25064 ~]# journalctl --no-pager --since="5 minutes ago" |grep '18:42:11' |grep -c clusterrole.rbac.authorization.k8s.io
Version-Release number of selected component (if applicable):
It's happening on all 3 masters in starter-us-east-1. I haven't seen it elsewhere.
Steps to Reproduce:
1. journalctl -fl
Many instances of this message scroll by every second:
Jan 06 18:48:16 ip-172-31-55-199.ec2.internal atomic-openshift-master-api: E0106 18:48:16.652886 16734 cache.go:332] error synchronizing: clusterrole.rbac.authorization.k8s.io "cluster-capacity-role" not found
There was a role binding, but no role. I have deleted the invalid role binding from the cluster in question to work around this BZ.
We should log, maybe even regularly. But not 850 times per second.
Eric I am sympatethic to the request, but how do you propose we handle this?
Should we write a new logging library that can deal with this ?
If the message needs to be logged, rate limit it yourself (write a helper so others can use it if it doesn't exist). If we don't really need to know about this message and/or if there is nothing the admin can or should do to correct the situtation, we should reduce the log level.
WIP upstream PR https://github.com/kubernetes/kubernetes/pull/58307
Do we have any update on this issue.
This has been fixed with https://github.com/openshift/origin/pull/21522
I have check the kube api log, no such log report now.
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.0.0-0.nightly-2019-03-06-074438 True False 43m Cluster version is 4.0.0-0.nightly-2019-03-06-074438
*** Bug 1699937 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.