Bug 1769832 - Unnecessarily high API server usage of the Node Tuning Operator's operand (openshift-tuned)
Summary: Unnecessarily high API server usage of the Node Tuning Operator's operand (op...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node Tuning Operator
Version: 4.2.z
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.2.z
Assignee: Jiří Mencák
QA Contact: Simon
Whiteboard: aos-scalability-42
Depends On: 1761920
Blocks: 1772802
TreeView+ depends on / blocked
Reported: 2019-11-07 15:05 UTC by Jiří Mencák
Modified: 2019-11-19 13:49 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1761920
: 1772802 (view as bug list)
Last Closed: 2019-11-19 13:49:01 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift openshift-tuned pull 32 0 'None' 'closed' 'Bug 1769832: [release-4.2] Improve scalability of openshift-tuned.' 2019-12-04 21:08:12 UTC
Red Hat Product Errata RHBA-2019:3869 0 None None None 2019-11-19 13:49:12 UTC

Description Jiří Mencák 2019-11-07 15:05:36 UTC
+++ This bug was initially created as a clone of Bug #1761920 +++

Description of problem:
openshift-tuned of the Node Tuning Operator issues unnecessary an expensive pod List() operation roughly every 60s.  The increased API usage becomes apparent (larger than any other system pod) at ~250 node count and fatal to an OpenShift cluster at ~1200 node count.

Version-Release number of selected component (if applicable):
Any 4.2 cluster.

How reproducible:
After scaling a cluster to ~250 nodes.

Steps to Reproduce:
1. Scale a cluster to ~250 node count and watch apiserver_request_count for the openshift-tuned pods.

Actual results:
openshift-tuned's apiserver_request_count higher than any other pod on the cluster.

Expected results:
Much lower apiserver_request_count for the openshift-tuned pod.

Additional info:

--- Additional comment from  on 2019-11-05 09:49:34 UTC ---

Tested https://github.com/openshift/openshift-tuned/pull/30 on a 15-node (4.3.0-0.nightly-2019-11-02-092336) cluster with 2000 openshift-tuned pods.  The PR seemed to have helped significantly and no excessive number of API calls was registered.

Also tested the same setup with the *unpatched* openshift-tuned (without PR30).  Excessive number of API calls became apparent at around 250 openshift-tuned pods and the cluster "died" after having instantiated 1800 openshift-tuned pods.  This hopefully resolves the scalability issue to at least 2k nodes in the short term.

--- Additional comment from Mike Fiedler on 2019-11-07 13:25:14 UTC ---

Marking verified in build 4.3.0-0.nightly-2019-11-02-092336 based on Jiri's testing in comment 1.  Further analysis will be performed during 4.3 large scale testing on Azure.

Comment 2 Simon 2019-11-14 20:16:43 UTC
Verified on 4.2.0-0.nightly-2019-11-13-203727
Haven't notice additional API calls.

Comment 3 Simon 2019-11-15 13:24:33 UTC
Tested with 15 nodes and 2000 openshift-tuned pods

Comment 5 errata-xmlrpc 2019-11-19 13:49:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.