Bug 1497326 - [RFE] Alert on >25% etcd memory usage on a master, take action
Summary: [RFE] Alert on >25% etcd memory usage on a master, take action
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.3.0
Assignee: Sam Batschelet
QA Contact: ge liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-29 19:23 UTC by Max Whittingham
Modified: 2023-09-14 04:09 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
We need to add a general tuning guide recommendation that etcd memory should never be allowed to grow past 25%. This is the doc to add it.
Clone Of:
Environment:
Last Closed: 2019-11-06 13:24:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Max Whittingham 2017-09-29 19:23:31 UTC 
Description of problem:
etcd timed out, lost connectivity to one of the members briefly, then reconnected.

Version-Release number of selected component (if applicable):
3.5.5.31-1

How reproducible:
I've not been able to reproduce this

Actual results:
Sep 29 19:14:04 $node.ec2.internal etcd[74733]: etcdserver: request timed out, possibly due to previous leader failure
Sep 29 19:14:06 $node.ec2.internal etcd[74733]: etcdserver: request timed out, possibly due to previous leader failure
Sep 29 19:14:11 $node.ec2.internal etcd[74733]: lost the TCP streaming connection with peer 3727844635f090bc (stream MsgApp v2 reader)

Expected results:
the etcdserver requests should not lose the streaming connection
Comment 5 Clayton Coleman 2017-10-03 18:44:54 UTC 
We need to alert on >25% etcd memory usage on a master and force an upgrade or prune.
Comment 9 Red Hat Bugzilla 2023-09-14 04:09:14 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.