Bug 1475558 - controller manager spam about PVCs
Summary: controller manager spam about PVCs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.7.0
Assignee: Avesh Agarwal
QA Contact: weiwei jiang
URL:
Whiteboard:
: 1487603 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-26 21:44 UTC by Eric Paris
Modified: 2017-11-28 22:05 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
This fix prevents unnecessary logging of a harmless error from a pvc related scheduler predicate.
Clone Of:
Environment:
Last Closed: 2017-11-28 22:05:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Eric Paris 2017-07-26 21:44:22 UTC
SO. MUCH. SPAM. We run as V(2).  Please either make this meaningful to an admin, remove it, or move it to a higher level.

oc v3.6.170

predicates.go:234] Unable to look up PVC info for dupa1/database, assuming PVC matches predicate when counting limits: persistentvolumeclaim "database" not found

Comment 1 Derek Carr 2017-07-27 14:20:07 UTC
scheduler log spam.

Comment 2 Derek Carr 2017-08-19 03:05:11 UTC
Avesh, any chance we can get a PR opened for this by 8/21 so it can make 3.6.1?

Comment 3 Avesh Agarwal 2017-08-22 16:01:28 UTC
(In reply to Derek Carr from comment #2)
> Avesh, any chance we can get a PR opened for this by 8/21 so it can make
> 3.6.1?

These errors are not returned as these are non-harmful errors. These errors are currently rate limited (not aggregated) in apimachinery/pkg/util/runtime/runtime.go. I think moving these to glog level 3 or beyond seems to be the only option to suppress these errors. Aggregation by returning these errors along with real errors seems hacky right now.

Comment 4 Jeremy Eder 2017-09-05 00:49:23 UTC
Aggregated across the 16 master nodes in the free and pro tiers, this log message is written over 1.2 million times in the last 24 hours (Sept 1).

Here are the top-talkers:

File and Line	               Lines Logged in 24 hours
--------------------------------------------------------
predicates.go line 234	              1282755
writers.go line 119	                 4035
status.go line 62	                 2805
factory.go line 641	                 2406
buildconfig_controller.go line 143       4587
proxier.go line 1320	                  727

Comment 5 Jeremy Eder 2017-09-05 00:49:40 UTC
*** Bug 1487603 has been marked as a duplicate of this bug. ***

Comment 6 Avesh Agarwal 2017-09-05 16:41:37 UTC
There is a PR to kube upstream for this by Matthew Wong@redhat: https://github.com/kubernetes/kubernetes/pull/48226.

Comment 10 Avesh Agarwal 2017-09-13 16:08:56 UTC
https://github.com/openshift/origin/pull/16329

Comment 13 DeShuai Ma 2017-09-30 09:49:03 UTC
Test on openshift v3.7.0-0.131.0, while the pod is running, pod's PVC is deleted or pods' PVC's PV is deleted. I don't seem the spam log, verify the bug.

Comment 16 errata-xmlrpc 2017-11-28 22:05:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.