Bug 1882030 - Traffic spikes every 10 minutes to master API
Summary: Traffic spikes every 10 minutes to master API
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.5.z
Assignee: Mike Dame
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On: 1854434
Blocks: 1882033
TreeView+ depends on / blocked
 
Reported: 2020-09-23 16:03 UTC by Mike Dame
Modified: 2020-10-19 14:55 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1854434
: 1882033 (view as bug list)
Environment:
Last Closed: 2020-10-19 14:54:54 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 25541 0 None closed Bug 1882030: UPSTREAM: 94112: Remove canonicalization of endpoints by endpoints controller for better comparison 2020-10-20 02:37:08 UTC
Red Hat Product Errata RHBA-2020:4228 0 None None None 2020-10-19 14:55:21 UTC

Comment 1 Mike Dame 2020-10-02 21:23:53 UTC 
Moving this to ON_QA, the PR merged and this changed to modified 5 days ago but never moved to ON_QA. If that's normal I apologize
Comment 2 RamaKasturi 2020-10-06 10:21:07 UTC 
verified with the payload below and i see that updates are lesser after bug fix has happened. Below are the steps followed to verify the bug.

[knarra@knarra ~]$ oc version
Client Version: 4.6.0-202009302026.p0-eadaf89
Server Version: 4.5.0-0.nightly-2020-10-05-204452
Kubernetes Version: v1.18.3+9cdca7b
[knarra@knarra ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-10-05-204452   True        False         4h53m   Cluster version is 4.5.0-0.nightly-2020-10-05-204452


steps followed to verify bug:
==============================
1) Install two 4.5 cluster one with payload before the fix, another with payload after the fix
2) Now set the loglevel of kubecontrollermanager operator to "Trace" on both the cluster

Before fix:
=========================
[knarra@knarra ~]$ oc get service --all-namespaces | wc -l
67

[knarra@knarra ~]$ oc logs pod/kube-controller-manager-ip-10-0-207-100.us-east-2.compute.internal -n openshift-kube-controller-manager -c kube-controller-manager | grep endpoints_controller | grep -c "About to update endpoints"
366
[knarra@knarra ~]$ oc logs pod/kube-controller-manager-ip-10-0-207-100.us-east-2.compute.internal -n openshift-kube-controller-manager -c kube-controller-manager | grep endpoints_controller | grep -c "Update endpoints for"
144

After fix:
========================
[knarra@knarra ~]$ oc get services --all-namespaces | wc -l
67

[knarra@knarra ~]$ oc logs pod/kube-controller-manager-ip-10-0-180-72.us-east-2.compute.internal -n openshift-kube-controller-manager -c kube-controller-manager | grep endpoints_controller | grep -c "About to update endpoints"
183
[knarra@knarra ~]$ oc logs pod/kube-controller-manager-ip-10-0-180-72.us-east-2.compute.internal -n openshift-kube-controller-manager -c kube-controller-manager | grep  endpoints_controller | grep -c "Update endpoints for"
0


Based on the above moving bug to verified state.
Comment 5 errata-xmlrpc 2020-10-19 14:54:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.15 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4228
Note You need to log in before you can comment on or make changes to this bug.