Bug 1772802 - Unnecessarily high API server usage of the Node Tuning Operator's operand (openshift-tuned)
Summary: Unnecessarily high API server usage of the Node Tuning Operator's operand (op...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node Tuning Operator
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.z
Assignee: jmencak
QA Contact: Simon
URL:
Whiteboard:
Depends On: 1761920 1769832
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-15 08:24 UTC by jmencak
Modified: 2019-12-17 07:17 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1769832
Environment:
Last Closed: 2019-12-17 07:17:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift openshift-tuned pull 33 'None' 'closed' 'Bug 1772802: Improve scalability of openshift-tuned.' 2019-12-03 07:00:39 UTC
Red Hat Product Errata RHBA-2019:4084 None None None 2019-12-17 07:17:15 UTC

Comment 2 Simon 2019-12-04 21:22:33 UTC
Verification positive!

Tested with 18 clusters, 2000 pods.

before fix
cluster version: 4.1.18
openshift-tuned v4.1.18-201909201915-0-gb176431-dirty

request count (prometheus: apiserver_request_count{client=~"openshift-tuned.+"}):

* Before test
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.140.17:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	507
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.140.17:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="LIST"}	507
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.140.17:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	2
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.159.162:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	145
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.159.162:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="LIST"}	145
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.167.244:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	597
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.167.244:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="LIST"}	597
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.167.244:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	3

sum: 2503

* After Test

apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.140.17:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	9327
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.140.17:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="LIST"}	9327
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.140.17:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	1701
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.159.162:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	8818
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.159.162:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="LIST"}	8818
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.159.162:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	1542
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.167.244:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	9159
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.167.244:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="LIST"}	9159
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.167.244:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	1628

sum: 59479

----

after fix
cluster version: 4.1.0-0.nightly-2019-12-04-071458
openshift-tuned v4.1.27-201912030019-0-g80617d4-dirty

request count (prometheus: apiserver_request_count{client=~"openshift-tuned.+"}):

* Before test

apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.136.47:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	19
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.145.237:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	24
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.163.190:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	38

sum: 81

* After test

apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.136.47:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	6433
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.136.47:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	1219
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.145.237:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	6614
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.145.237:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	1311
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.163.190:6443",job="apiserver",namespace="default",resource="nodes",scope="cluster",service="kubernetes",verb="GET"}	6384
apiserver_request_count{client="openshift-tuned/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.163.190:6443",job="apiserver",namespace="default",resource="pods",scope="cluster",service="kubernetes",verb="WATCH"}	1199

sum: 23160

Comment 3 Simon 2019-12-04 21:23:15 UTC
* 18 worker nodes :)

Comment 5 errata-xmlrpc 2019-12-17 07:17:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4084


Note You need to log in before you can comment on or make changes to this bug.