2103590 – [HyperShift] Election timeouts on OVNKube masters for Hypershift guests post statefulset recreation

Bug 2103590 - [HyperShift] Election timeouts on OVNKube masters for Hypershift guests post statefulset recreation

Summary: [HyperShift] Election timeouts on OVNKube masters for Hypershift guests post ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	4.12.0
Assignee:	Patryk Diak
QA Contact:	Ross Brattain
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	2096456
TreeView+	depends on / blocked

Reported:	2022-07-04 07:58 UTC by Patryk Diak
Modified:	2023-01-17 19:51 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:	2096456
Environment:
Last Closed:	2023-01-17 19:51:19 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-network-operator pull 1503	0	None	Merged	Bug 2103590: Add init container to ensure that Status.podIP is set before postStart hooks run	2022-07-11 18:40:17 UTC
Red Hat Product Errata	RHSA-2022:7399	0	None	None	None	2023-01-17 19:51:37 UTC

Comment 4 Ross Brattain 2022-07-19 03:13:32 UTC

Verified on 4.12.0-0.nightly-2022-07-17-215842


~30 seconds from end of deletion to new leader.

      pod "ovnkube-master-0" deleted
      pod "ovnkube-master-1" deleted
      pod "ovnkube-master-2" deleted
      [02:38:57] INFO> Exit Status: 0
      [02:39:29] INFO> cb.new_north_leader.name = ovnkube-master-2


      pod "ovnkube-master-0" deleted
      pod "ovnkube-master-1" deleted
      pod "ovnkube-master-2" deleted
      [02:46:12] INFO> Exit Status: 0
      [02:46:43] INFO> cb.new_north_leader.name = ovnkube-master-1


Pod election logs.

rg -e 'server \w+ is leader for term \d+' -e 'local server ID is \w+' -e 'elected leader by' -e 'learned server ID' -e '^\d+[^s]*starting [ns]bdb  CLUSTER_INITIATOR_IP' logs*

logs-3/log_ovnkube-master-1_ip-10-0-208-82.eu-central-1.compute.internal
126:2022-07-19T02:39:11+00:00 - starting nbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local, K8S_NODE_IP=10.0.208.82
177:2022-07-19T02:39:11.665Z|00002|raft|INFO|local server ID is 257e
190:2022-07-19T02:39:26.050Z|00013|raft|INFO|server cdad is leader for term 9
192:2022-07-19T02:39:28.494Z|00014|raft|INFO|ssl:10.128.2.54:50050: learned server ID cdad
194:2022-07-19T02:39:30.767Z|00016|raft|INFO|ssl:10.129.2.102:57026: learned server ID 644e
210:2022-07-19T02:39:28+00:00 - starting sbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local
273:2022-07-19T02:39:28.271Z|00002|raft|INFO|local server ID is 32d8
279:2022-07-19T02:39:29.119Z|00008|raft|INFO|ssl:10.128.2.54:44532: learned server ID 4d2f
281:2022-07-19T02:39:29.211Z|00010|raft|INFO|ssl:10.129.2.102:60690: learned server ID 1122
287:2022-07-19T02:39:44.921Z|00014|raft|INFO|server 1122 is leader for term 7

logs-3/log_ovnkube-master-2_ip-10-0-151-239.eu-central-1.compute.internal
117:2022-07-19T02:39:05+00:00 - starting nbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local, K8S_NODE_IP=10.0.151.239
168:2022-07-19T02:39:05.453Z|00002|raft|INFO|local server ID is cdad
187:2022-07-19T02:39:07.765Z|00019|raft|INFO|ssl:10.129.2.102:40876: learned server ID 644e
197:2022-07-19T02:39:11.703Z|00029|raft|INFO|ssl:10.131.0.68:59438: learned server ID 257e
217:2022-07-19T02:39:26.049Z|00049|raft|INFO|term 9: elected leader by 2+ of 3 servers
252:2022-07-19T02:39:27+00:00 - starting sbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local
315:2022-07-19T02:39:28.050Z|00002|raft|INFO|local server ID is 4d2f
321:2022-07-19T02:39:28.211Z|00008|raft|INFO|ssl:10.129.2.102:59822: learned server ID 1122
323:2022-07-19T02:39:28.338Z|00010|raft|INFO|ssl:10.131.0.68:37428: learned server ID 32d8
333:2022-07-19T02:39:44.921Z|00018|raft|INFO|server 1122 is leader for term 7

logs-3/log_ovnkube-master-0_ip-10-0-172-242.eu-central-1.compute.internal
102:2022-07-19T02:39:07+00:00 - starting nbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local, K8S_NODE_IP=10.0.172.242
153:2022-07-19T02:39:07.728Z|00002|raft|INFO|local server ID is 644e
171:2022-07-19T02:39:11.702Z|00018|raft|INFO|ssl:10.131.0.68:43348: learned server ID 257e
182:2022-07-19T02:39:20.491Z|00029|raft|INFO|ssl:10.128.2.54:47268: learned server ID cdad
185:2022-07-19T02:39:26.050Z|00032|raft|INFO|server cdad is leader for term 9
202:2022-07-19T02:39:27+00:00 - starting sbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local
265:2022-07-19T02:39:28.146Z|00002|raft|INFO|local server ID is 1122
271:2022-07-19T02:39:28.336Z|00008|raft|INFO|ssl:10.131.0.68:57762: learned server ID 32d8
273:2022-07-19T02:39:29.117Z|00010|raft|INFO|ssl:10.128.2.54:33742: learned server ID 4d2f
282:2022-07-19T02:39:44.921Z|00017|raft|INFO|term 7: elected leader by 2+ of 3 servers


log_ovnkube-master-1_ip-10-0-208-82.eu-central-1.compute.internal
109:2022-07-19T02:46:18+00:00 - starting nbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local, K8S_NODE_IP=10.0.208.82
160:2022-07-19T02:46:18.973Z|00002|raft|INFO|local server ID is 257e
187:2022-07-19T02:46:22.911Z|00027|raft|INFO|ssl:10.128.2.56:54150: learned server ID cdad
189:2022-07-19T02:46:25.977Z|00029|raft|INFO|ssl:10.129.2.105:46270: learned server ID 644e
207:2022-07-19T02:46:39.722Z|00047|raft|INFO|term 11: elected leader by 2+ of 3 servers
236:2022-07-19T02:46:41+00:00 - starting sbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local
299:2022-07-19T02:46:41.595Z|00002|raft|INFO|local server ID is 32d8
307:2022-07-19T02:46:42.585Z|00008|raft|INFO|ssl:10.128.2.56:59770: learned server ID 4d2f
309:2022-07-19T02:46:43.392Z|00010|raft|INFO|ssl:10.129.2.105:36048: learned server ID 1122
315:2022-07-19T02:46:57.804Z|00016|raft|INFO|term 9: elected leader by 2+ of 3 servers

log_ovnkube-master-0_ip-10-0-172-242.eu-central-1.compute.internal
105:2022-07-19T02:46:25+00:00 - starting nbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local, K8S_NODE_IP=10.0.172.242
156:2022-07-19T02:46:25.909Z|00002|raft|INFO|local server ID is 644e
169:2022-07-19T02:46:37.912Z|00013|raft|INFO|ssl:10.128.2.56:60882: learned server ID cdad
171:2022-07-19T02:46:39.723Z|00015|raft|INFO|server 257e is leader for term 11
173:2022-07-19T02:46:42.012Z|00016|raft|INFO|ssl:10.131.0.69:38860: learned server ID 257e
189:2022-07-19T02:46:40+00:00 - starting sbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local
252:2022-07-19T02:46:40.324Z|00002|raft|INFO|local server ID is 1122
266:2022-07-19T02:46:41.583Z|00014|raft|INFO|ssl:10.128.2.56:41716: learned server ID 4d2f
268:2022-07-19T02:46:41.665Z|00016|raft|INFO|ssl:10.131.0.69:40830: learned server ID 32d8
277:2022-07-19T02:46:57.804Z|00025|raft|INFO|server 32d8 is leader for term 9

log_ovnkube-master-2_ip-10-0-151-239.eu-central-1.compute.internal
18:2022-07-19T02:46:22+00:00 - starting nbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local, K8S_NODE_IP=10.0.151.239
69:2022-07-19T02:46:22.873Z|00002|raft|INFO|local server ID is cdad
87:2022-07-19T02:46:25.975Z|00018|raft|INFO|ssl:10.129.2.105:53220: learned server ID 644e
98:2022-07-19T02:46:34.012Z|00029|raft|INFO|ssl:10.131.0.69:53164: learned server ID 257e
101:2022-07-19T02:46:39.723Z|00032|raft|INFO|server 257e is leader for term 11
117:2022-07-19T02:46:41+00:00 - starting sbdb  CLUSTER_INITIATOR_IP=ovnkube-master-0.ovnkube-master-internal.clusters-hypershift-ci-23614.svc.cluster.local
180:2022-07-19T02:46:41.512Z|00002|raft|INFO|local server ID is 4d2f
186:2022-07-19T02:46:41.667Z|00008|raft|INFO|ssl:10.131.0.69:56624: learned server ID 32d8
192:2022-07-19T02:46:43.394Z|00012|raft|INFO|ssl:10.129.2.105:58714: learned server ID 1122
197:2022-07-19T02:46:57.805Z|00017|raft|INFO|server 32d8 is leader for term 9

Comment 7 errata-xmlrpc 2023-01-17 19:51:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399

Note You need to log in before you can comment on or make changes to this bug.