Bug 1997508 - cluster-version-operator pods logs flooded with "Throttling request took" messages
Summary: cluster-version-operator pods logs flooded with "Throttling request took" mes...
Keywords:
Status: CLOSED DUPLICATE of bug 2034493
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.6.z
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Lalatendu Mohanty
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-25 12:25 UTC by Robert Sandu
Modified: 2022-01-11 18:08 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-11 18:08:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Robert Sandu 2021-08-25 12:25:23 UTC
Description of problem:

As of [0] and [1], the CVO now runs in RHOCP 4.6 with `--v=5` log verbosity, which translates into 100 to 200k logs being generated daily by the CVO, most of which belong to:

1. The CVO syncing cluster manifests (i.e `sync_worker.go`).
2. The CVO logging `Throttling request took` messages (i.e. `request.go`).

Version-Release number of the following components: RHOCP 4.6 (GA and higher).

How reproducible: always

Steps to Reproduce:
1. standard CVO defaults in RHOCP 4.6 (GA and higher).

Actual results:
- 100 to 200k daily logs, putting additional disk I/O pressure on the cluster and Cluster Logging/Elasticsearch.

Expected results:
`Throttling request took` messages to not be logged.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1873900#c0
[1] https://github.com/openshift/cluster-version-operator/pull/448/commits/88c222c9549ffa3df00c9729f3dadb550edf2895

Comment 2 Lalatendu Mohanty 2021-08-31 16:05:38 UTC
@Robert Sandu Can you please attach the log file with the bug

Comment 5 Lalatendu Mohanty 2021-09-29 17:02:19 UTC
The code for this is in https://github.com/kubernetes/client-go/blob/1bccfc8c60977e2ce3235394daff128996818f7b/rest/request.go#L577 and not part of CVO code.

Comment 10 Lalatendu Mohanty 2022-01-11 18:08:42 UTC

*** This bug has been marked as a duplicate of bug 2034493 ***


Note You need to log in before you can comment on or make changes to this bug.