Bug 2078954

Summary:	network-metrics-daemon makes costly global pod list calls scaling per node
Product:	OpenShift Container Platform	Reporter:	Scott Dodson <sdodson>
Component:	Networking	Assignee:	Sebastian Scheinkman <sscheink>
Networking sub component:	multus	QA Contact:	Weibin Liang <weliang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	wking
Version:	4.8
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 11:08:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2064371, 2078988

Description Scott Dodson 2022-04-26 15:06:23 UTC

Description of problem:
Diagnosing a cluster failure recorded in Bug 2064371 it was observed that netowrk-metrics-daemon in multus makes a costly pod list call which inadvertently  does not benefit from apiserver caching mechanisms.

Version-Release number of selected component (if applicable):
4.8.17

How reproducible:
100%

Steps to Reproduce:
1. Scale to a significant number of pods, 17000 in the customer case
2. Observe network-metrics-daemon listing all pods in all namespaces on each node and apiserver / etcd resource consumption
3.

Actual results:
Costly pod list operations due to lack of RV

Expected results:
Less costly pod list operations leveraging watch cache

Additional info:
There are likely simpler synthetic tests which could be leveraged to test this more easily but that's beyond my understanding of the problem.

Comment 3 Weibin Liang 2022-05-12 14:45:48 UTC

Scale a cluster with 18177 pods, did't see large range reads on pods in 4.11.0-0.nightly-2022-05-11-054135

[weliang@weliang openshift-tests-private]$ oc logs etcd-ip-10-0-136-113.us-east-2.compute.internal -c etcd | grep range_response_count
{"level":"warn","ts":"2022-05-11T18:26:29.791Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"266.977935ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/openshift.io/rangeallocations/\" range_end:\"/openshift.io/rangeallocations0\" count_only:true ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"268.111756ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" serializable:true keys_only:true ","response":"range_response_count:1 size:53"}
{"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"243.489378ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" serializable:true keys_only:true ","response":"range_response_count:1 size:53"}
{"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"672.395652ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/operator.openshift.io/configs/cluster\" ","response":"range_response_count:1 size:2386"}
{"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"508.759724ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" serializable:true keys_only:true ","response":"range_response_count:1 size:53"}
{"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"649.798852ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/configmaps/openshift-etcd/etcd-scripts\" ","response":"range_response_count:1 size:12370"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"644.384228ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/pods/openshift-etcd/etcd-ip-10-0-193-92.us-east-2.compute.internal\" ","response":"range_response_count:1 size:47556"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"304.295249ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" keys_only:true ","response":"range_response_count:1 size:53"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"444.924475ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/migration.k8s.io/storageversionmigrations/\" range_end:\"/kubernetes.io/migration.k8s.io/storageversionmigrations0\" count_only:true ","response":"range_response_count:0 size:8"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"426.524533ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/roles/openshift-kube-apiserver-operator/prometheus-k8s\" ","response":"range_response_count:1 size:993"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"205.618438ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" keys_only:true ","response":"range_response_count:1 size:53"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"480.736924ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/health\" ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"241.343409ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/leases/kube-system/kube-controller-manager\" ","response":"range_response_count:1 size:518"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"344.230571ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/openshift.io/health\" ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"436.085987ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/configmaps/openshift-kube-controller-manager/trusted-ca-bundle\" ","response":"range_response_count:1 size:201158"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"627.217952ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" keys_only:true ","response":"range_response_count:1 size:53"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"280.776596ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/performance.openshift.io/performanceprofiles/\" range_end:\"/kubernetes.io/performance.openshift.io/performanceprofiles0\" count_only:true ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"266.425461ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/apiserver.openshift.io/apirequestcounts/clusterresourcequotas.v1.quota.openshift.io\" ","response":"range_response_count:1 size:10328"}
{"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"420.587575ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/pods/openshift-kube-apiserver/kube-apiserver-ip-10-0-136-113.us-east-2.compute.internal\" ","response":"range_response_count:1 size:17047"}
{"level":"warn","ts":"2022-05-11T21:06:58.675Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"368.48061ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/pods/\" range_end:\"/kubernetes.io/pods0\" count_only:true ","response":"range_response_count:0 size:9"}
[weliang@weliang openshift-tests-private]$

Comment 5 errata-xmlrpc 2022-08-10 11:08:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069