Bug 1882033 - Traffic spikes every 10 minutes to master API
Summary: Traffic spikes every 10 minutes to master API
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.4.z
Assignee: Mike Dame
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On: 1882030
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-23 16:04 UTC by Mike Dame
Modified: 2020-11-11 04:57 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1882030
Environment:
Last Closed: 2020-11-11 04:56:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 25543 0 None closed Bug 1882033: UPSTREAM: 94112: Remove canonicalization of endpoints by endpoints controller for better comparison 2020-11-11 04:57:49 UTC
Red Hat Product Errata RHBA-2020:4321 0 None None None 2020-11-11 04:57:01 UTC

Comment 1 Mike Dame 2020-10-02 21:24:45 UTC 
Adding upcomingsprint, but the work for this is done and just waiting for its dependent BZ to be verified so the changes can merge
Comment 3 RamaKasturi 2020-10-20 11:59:35 UTC 
verified with the payload below and i see that updates are less after the fix.

[knarra@knarra openshift-client-linux-4.4.0-0.nightly-2020-10-17-051433]$ ./oc version
Client Version: 4.4.0-0.nightly-2020-10-17-051433
Server Version: 4.4.0-0.nightly-2020-10-17-051433
Kubernetes Version: v1.17.1+5aced5f

set kubecontrollermanager loglevel to trace to be able to get the below results.

spec:
    logLevel: Trace

Before fix:
================
[knarra@knarra openshift-client-linux-4.4.27]$ ./oc get services --all-namespaces | wc -l
68
[knarra@knarra openshift-client-linux-4.4.27]$ ./oc logs kube-controller-manager-ip-10-0-197-171.us-east-2.compute.internal -n openshift-kube-controller-manager | grep endpoints_controller | grep -c "About to update endpoints"
806
[knarra@knarra openshift-client-linux-4.4.27]$ ./oc logs kube-controller-manager-ip-10-0-197-171.us-east-2.compute.internal -n openshift-kube-controller-manager | grep endpoints_controller | grep -c "Update endpoints for"
247

After fix:
=================
[knarra@knarra openshift-client-linux-4.4.0-0.nightly-2020-10-17-051433]$ ./oc get services --all-namespaces | wc -l
64
[knarra@knarra openshift-client-linux-4.4.0-0.nightly-2020-10-17-051433]$ ./oc logs kube-controller-manager-ip-10-0-146-14.us-east-2.compute.internal -n openshift-kube-controller-manager | grep endpoints_controller | grep -c "About to update endpoints"
696
[knarra@knarra openshift-client-linux-4.4.0-0.nightly-2020-10-17-051433]$ ./oc logs kube-controller-manager-ip-10-0-146-14.us-east-2.compute.internal -n openshift-kube-controller-manager | grep endpoints_controller | grep -c "Update endpoints for"
0

Based on the above moving bug to verified state.
Comment 6 errata-xmlrpc 2020-11-11 04:56:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.4.30 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4321
Note You need to log in before you can comment on or make changes to this bug.