Bug 1717497 - service-ca-operator leader election period is 2s causes excessive writing to etcd
Summary: service-ca-operator leader election period is 2s causes excessive writing to ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.0
Assignee: Matt Rogers
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-05 15:32 UTC by Clayton Coleman
Modified: 2019-10-16 06:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:31:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:31:26 UTC

Description Clayton Coleman 2019-06-05 15:32:54 UTC
service-ca-operator is using a 2s period for leader election

{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",resource="configmaps",verb="PUT"} 0.5

It should be using a much higher period, see defaults from library-go (I think we're at 15s or so).

Once fixed this should be back ported to 4.1 because it increases base load.

Comment 1 Michal Fojtik 2019-07-24 10:41:56 UTC
Seems to be already fixed in https://github.com/openshift/service-ca-operator/pull/55

Comment 2 Chuan Yu 2019-07-26 02:44:37 UTC
Verified on 4.2.0-0.nightly-2019-07-24-233914

{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",component="apiserver",contentType="application/json",endpoint="https",instance="10.0.130.163:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="GET",version="v1"}	0.1
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",component="apiserver",contentType="application/json",endpoint="https",instance="10.0.157.229:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="GET",version="v1"}	0.29630288080475864
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",component="apiserver",contentType="application/vnd.kubernetes.protobuf",endpoint="https",instance="10.0.130.163:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="GET",version="v1"}	0.044444444444444446


{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",component="apiserver",contentType="application/json",endpoint="https",instance="10.0.130.163:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="PUT",version="v1"}	0.1
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",component="apiserver",contentType="application/json",endpoint="https",instance="10.0.157.229:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="PUT",version="v1"}	0.3
{client="service-ca-operator/v0.0.0 (linux/amd64) kubernetes/$Format",code="200",component="apiserver",contentType="application/vnd.kubernetes.protobuf",endpoint="https",instance="10.0.157.229:6443",job="apiserver",namespace="default",resource="configmaps",scope="namespace",service="kubernetes",verb="PUT",version="v1"}	0.003703703703703704

Comment 5 errata-xmlrpc 2019-10-16 06:31:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.