Bug 1806932 - Two many "watch of *v1.Endpoints ended with: too old resource version" warn info in prometheus-k8s pod logs
Summary: Two many "watch of *v1.Endpoints ended with: too old resource version" warn i...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.5.0
Assignee: Paul Gier
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-25 10:29 UTC by Junqi Zhao
Modified: 2020-11-06 07:47 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-07 19:15:16 UTC
Target Upstream Version:


Attachments (Terms of Use)
prometheus-k8s pod logs (193.10 KB, text/plain)
2020-02-25 10:31 UTC, Junqi Zhao
no flags Details

Description Junqi Zhao 2020-02-25 10:29:32 UTC
Description of problem:
Two many "watch of *v1.Endpoints ended with: too old resource version" warn info in prometheus-k8s pod logs, the function is not affected
# oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0
level=warn ts=2020-02-25T08:38:51.231Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (202935)"
level=warn ts=2020-02-25T08:39:14.233Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203151)"
level=warn ts=2020-02-25T08:39:18.256Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203187)"
level=warn ts=2020-02-25T08:39:45.257Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203495)"
level=warn ts=2020-02-25T08:39:47.205Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203510)"
level=warn ts=2020-02-25T08:40:00.229Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203590)"

Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-02-24-105333

How reproducible:
always

Steps to Reproduce:
1. oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Junqi Zhao 2020-02-25 10:31:06 UTC
Created attachment 1665593 [details]
prometheus-k8s pod logs

Comment 3 Simon Pasquier 2020-02-25 15:05:17 UTC
This log message happens from time to time and there's nothing worrying about it. 

That being said, it's been fixed in the upstream k8s.io/client-go library not so long ago [1]. The next release of Prometheus will include the updated library and the log message will be logged at the info level so not visible unless Prometheus is started with --log.level=debug.

[1] https://github.com/kubernetes/kubernetes/pull/78991

Comment 4 Paul Gier 2020-04-07 19:15:16 UTC
Closing this since it will be fixed in the next release as Simon describes.


Note You need to log in before you can comment on or make changes to this bug.