Bug 2077975 - logging large range requests in < etcd-3.4.16 consumes significant memory
Summary: logging large range requests in < etcd-3.4.16 consumes significant memory
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.8.z
Assignee: melbeher
QA Contact: ge liu
URL:
Whiteboard:
Depends On: 2077501
Blocks: 2064371
TreeView+ depends on / blocked
 
Reported: 2022-04-22 18:49 UTC by Scott Dodson
Modified: 2022-10-04 10:27 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-08 13:31:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Scott Dodson 2022-04-22 18:49:38 UTC
Description of problem:
The logging function which produces the large range log messages unmarshals the content of the response when it need not do so. This bloats those requests by 10-50% based on some upstream testing. Fixing this could reduce memory consumption substantially on expensive requests and it's fixed upstream in 3.4.16.

Version-Release number of selected component (if applicable):
OCP 4.6-4.8, etcd-3.4.15 and earlier

See upstream issue and PR for more analysis
https://github.com/etcd-io/etcd/issues/12835
https://github.com/etcd-io/etcd/pull/12871

Since we should likely just update to the latest, 3.4.18 and that version contains fixes present in 3.5.2 but not 3.5.0 currently utilized by OCP 4.9 I will mark this as being dependent on the existing bugs which track updating 4.9 to 3.5.3. This ensures that we don't update 4.6-4.8 and then regress by upgrading into 4.9 which lacks the same fixes.

The logging memory consumption issue is already fixed in etcd 3.5.

Comment 2 melbeher 2022-05-12 12:52:49 UTC
Etcd 3.5.3 has been bumped into 4.9 release branch 

Does this suffice for this bug @sdodson

Comment 3 Scott Dodson 2022-05-12 13:52:06 UTC
Unless there's specific reason not to I'd like to see 4.6 through 4.8 bumped to the latest 3.4.z, at least 3.4.16 with the cited fix.

If it's too much work to get all those versions then 4.8 will suffice given we see most 4.6 and 4.7 clusters quickly upgrading to 4.8.

Comment 4 melbeher 2022-05-12 14:03:21 UTC
Ok, I will start with 4.8 and update here


Note You need to log in before you can comment on or make changes to this bug.