Bug 1806932

Summary: Two many "watch of *v1.Endpoints ended with: too old resource version" warn info in prometheus-k8s pod logs
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Paul Gier <pgier>
Status: CLOSED NEXTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.4CC: alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, naoto30, pkrupa, spasquie, surbania
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-07 19:15:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
prometheus-k8s pod logs none

Description Junqi Zhao 2020-02-25 10:29:32 UTC 
Description of problem:
Two many "watch of *v1.Endpoints ended with: too old resource version" warn info in prometheus-k8s pod logs, the function is not affected
# oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0
level=warn ts=2020-02-25T08:38:51.231Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (202935)"
level=warn ts=2020-02-25T08:39:14.233Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203151)"
level=warn ts=2020-02-25T08:39:18.256Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203187)"
level=warn ts=2020-02-25T08:39:45.257Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203495)"
level=warn ts=2020-02-25T08:39:47.205Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203510)"
level=warn ts=2020-02-25T08:40:00.229Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203590)"

Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-02-24-105333

How reproducible:
always

Steps to Reproduce:
1. oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Junqi Zhao 2020-02-25 10:31:06 UTC 
Created attachment 1665593 [details]
prometheus-k8s pod logs
Comment 3 Simon Pasquier 2020-02-25 15:05:17 UTC 
This log message happens from time to time and there's nothing worrying about it. 

That being said, it's been fixed in the upstream k8s.io/client-go library not so long ago [1]. The next release of Prometheus will include the updated library and the log message will be logged at the info level so not visible unless Prometheus is started with --log.level=debug.

[1] https://github.com/kubernetes/kubernetes/pull/78991
Comment 4 Paul Gier 2020-04-07 19:15:16 UTC 
Closing this since it will be fixed in the next release as Simon describes.