Bug 1462345 - controller manager spam about PVCs
controller manager spam about PVCs
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
3.5.1
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Avesh Agarwal
Weihua Meng
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-16 15:41 EDT by Eric Paris
Modified: 2017-08-16 15 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
With this fix, the error will include its message and its repeat count unlike before where the same error message was being output for each node in the cluster.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 01:28:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1716 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 05:02:50 EDT

  None (edit)
Description Eric Paris 2017-06-16 15:41:17 EDT
oc v3.5.5.19

/usr/bin/openshift start master controllers --config=/etc/origin/master/master-config.yaml --loglevel=2 --listen=https://0.0.0.0:8444

LOTS of messages in the logs every 30 seconds (might be more if not for systemd rate limitting)

Jun 16 18:42:22 ip-X-X-X-X.ec2.internal atomic-openshift-master-controllers[48569]: E0616 18:42:22.736961   48569 predicates.go:210] Unable to look up PVC info for mynamespace/mysql, assuming PVC matches predicate when counting limits: persistentvolumeclaims "mysql" not found

Jun 16 18:42:22 ip-172-31-54-162.ec2.internal atomic-openshift-master-controllers[48569]: E0616 18:42:22.975460   48569 factory.go:583] Error scheduling mynamespace mysql-1-4k4l7: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is une
Jun 16 18:42:22 ip-172-31-54-162.ec2.internal atomic-openshift-master-controllers[48569]: found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "
Comment 2 Derek Carr 2017-06-16 17:07:07 EDT
origin pr:
https://github.com/openshift/origin/pull/14714

need to verify that fit error is what is happening in eric's case.
Comment 3 Avesh Agarwal 2017-06-19 09:25:37 EDT
There are 2 things being discussed here:

1. Aggregation: My previous PR https://github.com/openshift/origin/pull/14495 addresses aggregation issue.

2. Suppression at default log level: For this, Derek's PR would need to be extended to include else part too that include error in question "Error scheduling".
Comment 4 Avesh Agarwal 2017-06-22 16:47:24 EDT
I am going to close this bug as next release. Please let me know if you disagree.
Comment 6 Weihua Meng 2017-06-29 05:38:09 EDT
I cannot reproduce it.
Could you show me detailed steps to reproduce it?
Thanks.
Comment 7 Eric Paris 2017-06-29 09:47:09 EDT
I have no idea how to reproduce. Marking 'NEEDSINFO' on Avesh. I just know that it was very common in the "starter" (aka online free tier) clusters.

Avesh, do you know how to reproduce?
Comment 8 Avesh Agarwal 2017-06-29 13:05:35 EDT
One way to reproduce it would be by tweaking the following line of code 

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/core/generic_scheduler.go#L228  

by hard coding err==nil.

Or otherwise, for example, for the following error:

"persistentvolumeclaims "mysql" not found"

try not to create any persistentvolumeclaims but somehow make the pod ask about it and see if that helps.
Comment 9 Weihua Meng 2017-06-29 23:24:25 EDT
Verified on openshift v3.6.126.4
Fixed.
create a pod with PVC, the PVC does not exist.
No spam message found.
Comment 11 errata-xmlrpc 2017-08-10 01:28:09 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.