Bug 1658895 - Renew lease for leader election every 2 seconds is too often
Summary: Renew lease for leader election every 2 seconds is too often
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Matthew Wong
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-13 06:19 UTC by Liang Xia
Modified: 2019-03-12 14:25 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-20 17:57:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Liang Xia 2018-12-13 06:19:16 UTC
Description of problem:
The CSI operator renewed lease every 2 seconds, which is too often.


Version-Release number of selected component (if applicable):
# oc version
oc v4.0.0-0.94.0
kubernetes v1.11.0+3db990d20d
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://preserve-qe-lxia-40-master-etcd-1:8443
openshift v4.0.0-0.95.0
kubernetes v1.11.0+8afe8f3cf9

image: registry.svc.ci.openshift.org/openshift/origin-v4.0:csi-operator
imageID: docker-pullable://registry.svc.ci.openshift.org/openshift/origin-v4.0@sha256:65257906a48860c744dc1cd7ba790fbed9ed17fe6491993551c51d7dcbace0ac


How reproducible:
Always


Steps to Reproduce:
1. Follow the steps to deploy csi-operator
https://github.com/openshift/csi-operator#usage
2. Check the operator pod.
$ oc -n openshift-csi-operator get pods -l name=csi-operator
$ oc -n openshift-csi-operator logs csi-operator-7b59d89cd-n4s4w


Actual results:
I1213 05:28:37.942181       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:39.956112       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:41.964160       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:43.972736       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:45.984318       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:47.992209       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:50.003927       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:52.013845       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:54.022425       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:56.029560       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:28:58.038457       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:29:00.049503       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:29:02.061933       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:29:04.068762       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader
I1213 05:29:06.078675       1 leaderelection.go:209] successfully renewed lease openshift-csi-operator/csi-operator-leader


Expected results:
Renew happended in a reasonable period.


Additional info:
# oc -n openshift-csi-operator get cm csi-operator-leader -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"csi-operator-7b59d89cd-n4s4w_1ae867f7-fe7f-11e8-90cc-0a580a82000d","leaseDurationSeconds":15,"acquireTime":"2018-12-13T02:30:50Z","renewTime":"2018-12-13T05:41:27Z","leaderTransitions":0}'
  creationTimestamp: 2018-12-13T02:30:50Z
  name: csi-operator-leader
  namespace: openshift-csi-operator
  resourceVersion: "214672"
  selfLink: /api/v1/namespaces/openshift-csi-operator/configmaps/csi-operator-leader
  uid: 1b42b475-fe7f-11e8-aff4-fa163e476db5

Comment 1 Matthew Wong 2018-12-20 17:57:52 UTC
The 2s interval is default determined by controller-runtime & is not configurable: https://github.com/kubernetes-sigs/controller-runtime/blob/a4051585e748b437e799d597f9adba1d777cb12a/pkg/manager/internal.go#L271. 2s should be fine, but logging every 2s is excessive. Logging verbosity will be fixed when operator-sdk updates to client-go 1.13 https://github.com/kubernetes/client-go/commit/74d0afcfd9ce26d9878915d75ecbfc7e80038a51#diff-d8c539591cadece67f3478771221342c which may not be for a while (4.1?).


Note You need to log in before you can comment on or make changes to this bug.