+++ This bug was initially created as a clone of Bug #1761920 +++ Description of problem: openshift-tuned of the Node Tuning Operator issues unnecessary an expensive pod List() operation roughly every 60s. The increased API usage becomes apparent (larger than any other system pod) at ~250 node count and fatal to an OpenShift cluster at ~1200 node count. Version-Release number of selected component (if applicable): Any 4.2 cluster. How reproducible: After scaling a cluster to ~250 nodes. Steps to Reproduce: 1. Scale a cluster to ~250 node count and watch apiserver_request_count for the openshift-tuned pods. Actual results: openshift-tuned's apiserver_request_count higher than any other pod on the cluster. Expected results: Much lower apiserver_request_count for the openshift-tuned pod. Additional info: https://github.com/openshift/openshift-tuned/pull/30 --- Additional comment from on 2019-11-05 09:49:34 UTC --- Tested https://github.com/openshift/openshift-tuned/pull/30 on a 15-node (4.3.0-0.nightly-2019-11-02-092336) cluster with 2000 openshift-tuned pods. The PR seemed to have helped significantly and no excessive number of API calls was registered. Also tested the same setup with the *unpatched* openshift-tuned (without PR30). Excessive number of API calls became apparent at around 250 openshift-tuned pods and the cluster "died" after having instantiated 1800 openshift-tuned pods. This hopefully resolves the scalability issue to at least 2k nodes in the short term. --- Additional comment from Mike Fiedler on 2019-11-07 13:25:14 UTC --- Marking verified in build 4.3.0-0.nightly-2019-11-02-092336 based on Jiri's testing in comment 1. Further analysis will be performed during 4.3 large scale testing on Azure.
Verified on 4.2.0-0.nightly-2019-11-13-203727 Haven't notice additional API calls.
Tested with 15 nodes and 2000 openshift-tuned pods
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3869