Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1834474

Summary:	[4.4] ovnkube: set NB/SB database inactivity probes to 60 seconds
Product:	OpenShift Container Platform	Reporter:	Dan Williams <dcbw>
Component:	Networking	Assignee:	Ricardo Carrillo Cruz <ricarril>
Networking sub component:	ovn-kubernetes	QA Contact:	Ross Brattain <rbrattai>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	bbennett, ingvarr.zhmakin, zzhao
Version:	4.3.z
Target Milestone:	---
Target Release:	4.4.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1834473	Environment:
Last Closed:	2020-06-23 00:57:24 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1834473
Bug Blocks:

Description Dan Williams 2020-05-11 19:07:32 UTC

+++ This bug was initially created as a clone of Bug #1834473 +++

Multiple northds run for HA in active/passive mode where the active
northd holds a lock. If that northd loses connectivity to the database
or is killed without releasing the lock, ovsdb-server will clear
the lock after twice the inactivity probe. But if that probe is set
to 0 (disabled) that will never happen, and a new northd will never
grab the lock and continue reconciling NB->SB.

Set the DB inactivity probe to something greater than 0 to ensure
that a northd will always eventually become active. The value of 60 was
chosen as a reasonable middle-ground between the lock being cleared
and another northd grabbing it (~120s) and the possibility that a loaded
ovsdb-server (many ovn-controller clients) would take more than 30-40
seconds to send/reply to all inactivity probes from clients.

Comment 3 Ross Brattain 2020-06-17 05:56:48 UTC

Verified on 4.4.0-0.nightly-2020-06-15-203705

w9h72 has lock

northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:40Z|10355|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:41Z|10372|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:54Z|10429|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"

w9h72: pkill -STOP ovn-northd

ccd5d takes lock ~120 seconds later

northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14663|jsonrpc|DBG|ssl:10.0.195.93:9642: received notification, method="locked", params=["ovn_northd"]
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14664|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14674|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:43:10Z|14739|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:43:11Z|14754|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"

Comment 5 errata-xmlrpc 2020-06-23 00:57:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2580