Description of problem: openshift-tuned of the Node Tuning Operator issues unnecessary an expensive pod List() operation roughly every 60s. The increased API usage becomes apparent (larger than any other system pod) at ~250 node count and fatal to an OpenShift cluster at ~1200 node count. Version-Release number of selected component (if applicable): Any 4.2 cluster. How reproducible: After scaling a cluster to ~250 nodes. Steps to Reproduce: 1. Scale a cluster to ~250 node count and watch apiserver_request_count for the openshift-tuned pods. Actual results: openshift-tuned's apiserver_request_count higher than any other pod on the cluster. Expected results: Much lower apiserver_request_count for the openshift-tuned pod. Additional info: https://github.com/openshift/openshift-tuned/pull/30
Tested https://github.com/openshift/openshift-tuned/pull/30 on a 15-node (4.3.0-0.nightly-2019-11-02-092336) cluster with 2000 openshift-tuned pods. The PR seemed to have helped significantly and no excessive number of API calls was registered. Also tested the same setup with the *unpatched* openshift-tuned (without PR30). Excessive number of API calls became apparent at around 250 openshift-tuned pods and the cluster "died" after having instantiated 1800 openshift-tuned pods. This hopefully resolves the scalability issue to at least 2k nodes in the short term.
Marking verified in build 4.3.0-0.nightly-2019-11-02-092336 based on Jiri's testing in comment 1. Further analysis will be performed during 4.3 large scale testing on Azure.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1529