Bug 1997508

Summary:	cluster-version-operator pods logs flooded with "Throttling request took" messages
Product:	OpenShift Container Platform	Reporter:	Robert Sandu <rsandu>
Component:	Cluster Version Operator	Assignee:	Lalatendu Mohanty <lmohanty>
Status:	CLOSED DUPLICATE	QA Contact:	Johnny Liu <jialiu>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.6.z	CC:	aos-bugs, bleanhar, lmohanty, pescorza, rsandu, scott.worthington, wking, yanyang
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-01-11 18:08:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Robert Sandu 2021-08-25 12:25:23 UTC

Description of problem:

As of [0] and [1], the CVO now runs in RHOCP 4.6 with `--v=5` log verbosity, which translates into 100 to 200k logs being generated daily by the CVO, most of which belong to:

1. The CVO syncing cluster manifests (i.e `sync_worker.go`).
2. The CVO logging `Throttling request took` messages (i.e. `request.go`).

Version-Release number of the following components: RHOCP 4.6 (GA and higher).

How reproducible: always

Steps to Reproduce:
1. standard CVO defaults in RHOCP 4.6 (GA and higher).

Actual results:
- 100 to 200k daily logs, putting additional disk I/O pressure on the cluster and Cluster Logging/Elasticsearch.

Expected results:
`Throttling request took` messages to not be logged.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1873900#c0
[1] https://github.com/openshift/cluster-version-operator/pull/448/commits/88c222c9549ffa3df00c9729f3dadb550edf2895

Comment 2 Lalatendu Mohanty 2021-08-31 16:05:38 UTC

@Robert Sandu Can you please attach the log file with the bug

Comment 5 Lalatendu Mohanty 2021-09-29 17:02:19 UTC

The code for this is in https://github.com/kubernetes/client-go/blob/1bccfc8c60977e2ce3235394daff128996818f7b/rest/request.go#L577 and not part of CVO code.

Comment 10 Lalatendu Mohanty 2022-01-11 18:08:42 UTC


*** This bug has been marked as a duplicate of bug 2034493 ***