Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1761920

Summary:	Unnecessarily high API server usage of the Node Tuning Operator's operand (openshift-tuned)
Product:	OpenShift Container Platform	Reporter:	Jiří Mencák <jmencak>
Component:	Node Tuning Operator	Assignee:	Jiří Mencák <jmencak>
Status:	CLOSED ERRATA	QA Contact:	Simon <skordas>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	4.3.0	CC:	mifiedle, nelluri, sejug, sreber
Target Milestone:	---
Target Release:	4.3.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	aos-scalability-42
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	1769832 (view as bug list)		Environment:
Last Closed:	2020-04-30 01:28:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1769832, 1772802

Description Jiří Mencák 2019-10-15 15:09:16 UTC

Description of problem:
openshift-tuned of the Node Tuning Operator issues unnecessary an expensive pod List() operation roughly every 60s.  The increased API usage becomes apparent (larger than any other system pod) at ~250 node count and fatal to an OpenShift cluster at ~1200 node count.

Version-Release number of selected component (if applicable):
Any 4.2 cluster.

How reproducible:
After scaling a cluster to ~250 nodes.

Steps to Reproduce:
1. Scale a cluster to ~250 node count and watch apiserver_request_count for the openshift-tuned pods.

Actual results:
openshift-tuned's apiserver_request_count higher than any other pod on the cluster.

Expected results:
Much lower apiserver_request_count for the openshift-tuned pod.

Additional info:
https://github.com/openshift/openshift-tuned/pull/30

Comment 1 Jiří Mencák 2019-11-05 09:49:34 UTC

Tested https://github.com/openshift/openshift-tuned/pull/30 on a 15-node (4.3.0-0.nightly-2019-11-02-092336) cluster with 2000 openshift-tuned pods.  The PR seemed to have helped significantly and no excessive number of API calls was registered.

Also tested the same setup with the *unpatched* openshift-tuned (without PR30).  Excessive number of API calls became apparent at around 250 openshift-tuned pods and the cluster "died" after having instantiated 1800 openshift-tuned pods.  This hopefully resolves the scalability issue to at least 2k nodes in the short term.

Comment 2 Mike Fiedler 2019-11-07 13:25:14 UTC

Marking verified in build 4.3.0-0.nightly-2019-11-02-092336 based on Jiri's testing in comment 1.  Further analysis will be performed during 4.3 large scale testing on Azure.

Comment 5 errata-xmlrpc 2020-04-30 01:28:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1529