Bug 1466933 - Spam to API server is causing too many etcd writes
Spam to API server is causing too many etcd writes
Status: VERIFIED
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.7.0
Assigned To: Derek Carr
Mike Fiedler
:
Depends On:
Blocks: 1467022
  Show dependency treegraph
 
Reported: 2017-06-30 15:34 EDT by Derek Carr
Modified: 2017-09-25 10:12 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: control event spam to masters Reason: internal controllers can send large numbers of events when an object is unable to reach its desired state. in large clusters, this can cause excessive writes to etcd. Result: event client has been updated to protect against spamming master components. objects have an initial event budget of 25 events with a refill rate of 1 event every 5 minutes. this controls traffic to the masters and reduces writes to etcd.
Story Points: ---
Clone Of:
: 1467022 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Derek Carr 2017-06-30 15:34:03 EDT
Description of problem:

A large cluster with either of the following:

1. a large amount of HPAs
2. a large amount of pods that are always unhealthy

is causing too many writes to the API server and inducing excessive snapshot of etcd.

Version-Release number of selected component (if applicable):
OCP 3.6.0

How reproducible:
Always

Steps to Reproduce:
See above

Actual results:
spam should be reduced to masters

Expected results:
HPA should only send status updates if status changes
Pods in perpetual failure status should not spam masters with events

Additional info:
Comment 1 Derek Carr 2017-06-30 16:28:27 EDT
Two fixes were identified:

https://github.com/openshift/origin/pull/14747

this sets a budget of events about an object.
per source+object event budget of 25 burst with refill of 1 every 5 minutes.
it will reduce the long tail of events sent about objects in perpetual failure states (pod in crash loop backoff, controllers denied by quota, etc.).

https://github.com/openshift/origin/pull/14529
HPA only sends status updates if status changes.
Comment 2 Eric Paris 2017-07-01 16:14:14 EDT
I'm using this bz to track only 
https://github.com/openshift/origin/pull/14747

The HPA fix is tracked in 1467022
Comment 4 DeShuai Ma 2017-07-05 04:06:22 EDT
Hi Mike, help verify the bug, thanks.
Comment 5 Mike Fiedler 2017-07-06 17:01:00 EDT
Verified during an event storm caused by misconfigured DNS.  api servers were not getting unique PUTs for each event recorded by the kubelet.   v3.6.126.1

Note You need to log in before you can comment on or make changes to this bug.