Bug 1703229 - High write rate from operator updating config maps, needs to be fixed
Summary: High write rate from operator updating config maps, needs to be fixed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.0
Assignee: Matt Rogers
QA Contact: Chuan Yu
URL:
Whiteboard: If this bug is not fixed by Friday, M...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-25 20:36 UTC by Clayton Coleman
Modified: 2019-06-04 10:48 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:48:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:48:12 UTC

Description Clayton Coleman 2019-04-25 20:36:07 UTC
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",resource="configmaps",scope="namespace",verb="GET"}	1.6111111111111112
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",resource="configmaps",scope="namespace",verb="PUT"}	1.5370370370370372

This is reads and writes per second, it looks like a hot loop.  Must fix for GA because it impacts total write load on cluster.

Comment 2 Vikas Choudhary 2019-04-29 08:48:13 UTC
Looks like the same issue: https://bugzilla.redhat.com/show_bug.cgi?id=1703232

Working to fix it.

Comment 4 Vikas Choudhary 2019-04-29 13:20:16 UTC
Currently controller-runtime repo runs leader election with hard-coded, very agressive values. With above PRs, leader election configuration will become configurable. Then we would pass higher time durations from cluster-autoscaler-operator using the options which above PRs are adding.

Comment 5 Mo 2019-04-29 14:10:12 UTC
(In reply to Vikas Choudhary from comment #4)
> Currently controller-runtime repo runs leader election with hard-coded, very
> agressive values. With above PRs, leader election configuration will become
> configurable. Then we would pass higher time durations from
> cluster-autoscaler-operator using the options which above PRs are adding.

Service CA does not use controller-runtime

Comment 9 Chuan Yu 2019-05-05 07:38:22 UTC
Verified on 4.1.0-0.nightly-2019-05-04-210601.

{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.148.133:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="GET"}	0.1
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.148.133:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="PUT"}	0.1
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.172.173:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="GET"}	0.3
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",contentType="application/json",endpoint="https",instance="10.0.172.173:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="PUT"}	0.3

Comment 11 errata-xmlrpc 2019-06-04 10:48:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.