Bug 1754788

Summary: Master-controllers log spammed with "The resourceVersion for the provided watch is too old." warning messages"
Product: OpenShift Container Platform Reporter: K Chandra Sekar <csekar>
Component: kube-controller-managerAssignee: Sally <somalley>
Status: CLOSED ERRATA QA Contact: zhou ying <yinzhou>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, mfojtik
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-18 01:34:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description K Chandra Sekar 2019-09-24 05:44:59 UTC
Description of problem:

master-controllers log spammed with "The resourceVersion for the provided watch is too old." warning messages. 

This warning messages are converted to Information messages on OCP v3.9 with the errata[1] created in the bugzilla[2] but we can see still see these messages are listed as Warning and not as Information in OCP v3.11 

~~~~~
oc logs pod/master-controllers-xxx

W0919 11:23:34.983377       1 reflector.go:272] k8s.io/client-go/informers/factory.go:130: watch of *v1.PersistentVolumeClaim ended with: The resourceVersion for the provided watch is too old.
W0919 11:23:48.902273       1 reflector.go:272] k8s.io/client-go/informers/factory.go:130: watch of *v1.StorageClass ended with: The resourceVersion for the provided watch is too old.
W0919 11:30:16.382786       1 reflector.go:272] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.StatefulSet ended with: The resourceVersion for the provided watch is too old.
W0919 11:32:04.568718       1 reflector.go:272] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.ReplicaSet ended with: The resourceVersion for the provided watch is too old.

~~~~~

[1] - https://access.redhat.com/errata/RHBA-2019:2550

[2] - https://bugzilla.redhat.com/show_bug.cgi?id=1573460


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
Check the controller pod logs in kube-system namespace to witness these logs

Actual results: The message "The resourceVersion for the provided watch is too old" is listed as a Warning Message.


Expected results: The message "The resourceVersion for the provided watch is too old" should be a Informational message not a Warning Message.


Additional info:

Comment 1 Maciej Szulik 2019-09-24 08:16:04 UTC
Sally you were working on https://bugzilla.redhat.com/show_bug.cgi?id=1573460 and picked https://github.com/kubernetes/kubernetes/pull/78991 into 3.9 in https://github.com/openshift/origin/pull/23284,
mind re-picking this into origin 3.11

Comment 5 zhou ying 2019-10-12 08:29:24 UTC
Confirmed with version, the issue has fixed:
[root@ip-172-18-10-58 ~]# oc version
oc v3.11.153
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-10-58.ec2.internal:8443
openshift v3.11.153
kubernetes v1.11.0+d4cacc0


[root@ip-172-18-10-58 ~]# oc logs -f master-controllers-xxxxx |grep "The resourceVersion for the provided watch"
I1012 04:10:49.317530       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1.ResourceQuota ended with: The resourceVersion for the provided watch is too old.
I1012 04:11:29.481605       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.Ingress ended with: The resourceVersion for the provided watch is too old.
I1012 04:11:35.296693       1 reflector.go:274] github.com/openshift/client-go/build/informers/externalversions/factory.go:101: watch of *v1.Build ended with: The resourceVersion for the provided watch is too old.
I1012 04:11:41.212701       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.CronJob ended with: The resourceVersion for the provided watch is too old.
I1012 04:12:03.398060       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.PodSecurityPolicy ended with: The resourceVersion for the provided watch is too old.
I1012 04:12:18.470360       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.VolumeAttachment ended with: The resourceVersion for the provided watch is too old.
I1012 04:12:23.294926       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1.HorizontalPodAutoscaler ended with: The resourceVersion for the provided watch is too old.
I1012 04:12:33.033484       1 reflector.go:274] k8s.io/client-go/informers/factory.go:130: watch of *v1beta1.PodDisruptionBudget ended with: The resourceVersion for the provided watch is too old.
I1012 04:12:34.241481       1 reflector.go:274] github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:101: watch of *quota.ClusterResourceQuota ended with: The resourceVersion for the provided watch is too old.

Comment 7 errata-xmlrpc 2019-10-18 01:34:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3139