Description of problem: Diagnosing a cluster failure recorded in Bug 2064371 it was observed that netowrk-metrics-daemon in multus makes a costly pod list call which inadvertently does not benefit from apiserver caching mechanisms. Version-Release number of selected component (if applicable): 4.8.17 How reproducible: 100% Steps to Reproduce: 1. Scale to a significant number of pods, 17000 in the customer case 2. Observe network-metrics-daemon listing all pods in all namespaces on each node and apiserver / etcd resource consumption 3. Actual results: Costly pod list operations due to lack of RV Expected results: Less costly pod list operations leveraging watch cache Additional info: There are likely simpler synthetic tests which could be leveraged to test this more easily but that's beyond my understanding of the problem.
Scale a cluster with 18177 pods, did't see large range reads on pods in 4.11.0-0.nightly-2022-05-11-054135 [weliang@weliang openshift-tests-private]$ oc logs etcd-ip-10-0-136-113.us-east-2.compute.internal -c etcd | grep range_response_count {"level":"warn","ts":"2022-05-11T18:26:29.791Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"266.977935ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/openshift.io/rangeallocations/\" range_end:\"/openshift.io/rangeallocations0\" count_only:true ","response":"range_response_count:0 size:6"} {"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"268.111756ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" serializable:true keys_only:true ","response":"range_response_count:1 size:53"} {"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"243.489378ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" serializable:true keys_only:true ","response":"range_response_count:1 size:53"} {"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"672.395652ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/operator.openshift.io/configs/cluster\" ","response":"range_response_count:1 size:2386"} {"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"508.759724ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" serializable:true keys_only:true ","response":"range_response_count:1 size:53"} {"level":"warn","ts":"2022-05-11T21:06:58.672Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"649.798852ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/configmaps/openshift-etcd/etcd-scripts\" ","response":"range_response_count:1 size:12370"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"644.384228ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/pods/openshift-etcd/etcd-ip-10-0-193-92.us-east-2.compute.internal\" ","response":"range_response_count:1 size:47556"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"304.295249ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" keys_only:true ","response":"range_response_count:1 size:53"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"444.924475ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/migration.k8s.io/storageversionmigrations/\" range_end:\"/kubernetes.io/migration.k8s.io/storageversionmigrations0\" count_only:true ","response":"range_response_count:0 size:8"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"426.524533ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/roles/openshift-kube-apiserver-operator/prometheus-k8s\" ","response":"range_response_count:1 size:993"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"205.618438ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" keys_only:true ","response":"range_response_count:1 size:53"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"480.736924ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/health\" ","response":"range_response_count:0 size:6"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"241.343409ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/leases/kube-system/kube-controller-manager\" ","response":"range_response_count:1 size:518"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"344.230571ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/openshift.io/health\" ","response":"range_response_count:0 size:6"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"436.085987ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/configmaps/openshift-kube-controller-manager/trusted-ca-bundle\" ","response":"range_response_count:1 size:201158"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"627.217952ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/namespaces/default\" keys_only:true ","response":"range_response_count:1 size:53"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"280.776596ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/performance.openshift.io/performanceprofiles/\" range_end:\"/kubernetes.io/performance.openshift.io/performanceprofiles0\" count_only:true ","response":"range_response_count:0 size:6"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"266.425461ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/apiserver.openshift.io/apirequestcounts/clusterresourcequotas.v1.quota.openshift.io\" ","response":"range_response_count:1 size:10328"} {"level":"warn","ts":"2022-05-11T21:06:58.674Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"420.587575ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/pods/openshift-kube-apiserver/kube-apiserver-ip-10-0-136-113.us-east-2.compute.internal\" ","response":"range_response_count:1 size:17047"} {"level":"warn","ts":"2022-05-11T21:06:58.675Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"368.48061ms","expected-duration":"200ms","prefix":"read-only range ","request":"key:\"/kubernetes.io/pods/\" range_end:\"/kubernetes.io/pods0\" count_only:true ","response":"range_response_count:0 size:9"} [weliang@weliang openshift-tests-private]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069