Bug 1462345

Summary:	controller manager spam about PVCs
Product:	OpenShift Container Platform	Reporter:	Eric Paris <eparis>
Component:	Node	Assignee:	Avesh Agarwal <avagarwa>
Status:	CLOSED ERRATA	QA Contact:	Weihua Meng <wmeng>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.5.1	CC:	aos-bugs, avagarwa, decarr, eparis, jokerman, mmccomas, smunilla, wmeng
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	With this fix, the error will include its message and its repeat count unlike before where the same error message was being output for each node in the cluster.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-10 05:28:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eric Paris 2017-06-16 19:41:17 UTC

oc v3.5.5.19

/usr/bin/openshift start master controllers --config=/etc/origin/master/master-config.yaml --loglevel=2 --listen=https://0.0.0.0:8444

LOTS of messages in the logs every 30 seconds (might be more if not for systemd rate limitting)

Jun 16 18:42:22 ip-X-X-X-X.ec2.internal atomic-openshift-master-controllers[48569]: E0616 18:42:22.736961   48569 predicates.go:210] Unable to look up PVC info for mynamespace/mysql, assuming PVC matches predicate when counting limits: persistentvolumeclaims "mysql" not found

Jun 16 18:42:22 ip-172-31-54-162.ec2.internal atomic-openshift-master-controllers[48569]: E0616 18:42:22.975460   48569 factory.go:583] Error scheduling mynamespace mysql-1-4k4l7: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is une
Jun 16 18:42:22 ip-172-31-54-162.ec2.internal atomic-openshift-master-controllers[48569]: found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "

Comment 2 Derek Carr 2017-06-16 21:07:07 UTC

origin pr:
https://github.com/openshift/origin/pull/14714

need to verify that fit error is what is happening in eric's case.

Comment 3 Avesh Agarwal 2017-06-19 13:25:37 UTC

There are 2 things being discussed here:

1. Aggregation: My previous PR https://github.com/openshift/origin/pull/14495 addresses aggregation issue.

2. Suppression at default log level: For this, Derek's PR would need to be extended to include else part too that include error in question "Error scheduling".

Comment 4 Avesh Agarwal 2017-06-22 20:47:24 UTC

I am going to close this bug as next release. Please let me know if you disagree.

Comment 6 Weihua Meng 2017-06-29 09:38:09 UTC

I cannot reproduce it.
Could you show me detailed steps to reproduce it?
Thanks.

Comment 7 Eric Paris 2017-06-29 13:47:09 UTC

I have no idea how to reproduce. Marking 'NEEDSINFO' on Avesh. I just know that it was very common in the "starter" (aka online free tier) clusters.

Avesh, do you know how to reproduce?

Comment 8 Avesh Agarwal 2017-06-29 17:05:35 UTC

One way to reproduce it would be by tweaking the following line of code 

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/core/generic_scheduler.go#L228  

by hard coding err==nil.

Or otherwise, for example, for the following error:

"persistentvolumeclaims "mysql" not found"

try not to create any persistentvolumeclaims but somehow make the pod ask about it and see if that helps.

Comment 9 Weihua Meng 2017-06-30 03:24:25 UTC

Verified on openshift v3.6.126.4
Fixed.
create a pod with PVC, the PVC does not exist.
No spam message found.

Comment 11 errata-xmlrpc 2017-08-10 05:28:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716