Description of problem: Two many "watch of *v1.Endpoints ended with: too old resource version" warn info in prometheus-k8s pod logs, the function is not affected # oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0 level=warn ts=2020-02-25T08:38:51.231Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (202935)" level=warn ts=2020-02-25T08:39:14.233Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203151)" level=warn ts=2020-02-25T08:39:18.256Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203187)" level=warn ts=2020-02-25T08:39:45.257Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203495)" level=warn ts=2020-02-25T08:39:47.205Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203510)" level=warn ts=2020-02-25T08:40:00.229Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 199192 (203590)" Version-Release number of selected component (if applicable): 4.4.0-0.nightly-2020-02-24-105333 How reproducible: always Steps to Reproduce: 1. oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0 2. 3. Actual results: Expected results: Additional info:
Created attachment 1665593 [details] prometheus-k8s pod logs
This log message happens from time to time and there's nothing worrying about it. That being said, it's been fixed in the upstream k8s.io/client-go library not so long ago [1]. The next release of Prometheus will include the updated library and the log message will be logged at the info level so not visible unless Prometheus is started with --log.level=debug. [1] https://github.com/kubernetes/kubernetes/pull/78991
Closing this since it will be fixed in the next release as Simon describes.