Bug 1834474 - [4.4] ovnkube: set NB/SB database inactivity probes to 60 seconds
Summary: [4.4] ovnkube: set NB/SB database inactivity probes to 60 seconds
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.4.z
Assignee: Ricardo Carrillo Cruz
QA Contact: Ross Brattain
Depends On: 1834473
TreeView+ depends on / blocked
Reported: 2020-05-11 19:07 UTC by Dan Williams
Modified: 2020-06-23 00:57 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1834473
Last Closed: 2020-06-23 00:57:24 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-network-operator pull 636 0 None closed Bug 1834474: ovnkube: set NB/SB database inactivity probes to 60 seconds 2020-10-11 17:00:36 UTC
Github openshift cluster-network-operator pull 646 0 None closed Bug 1834474: set NB/SB database inactivity probes to 60 seconds 2020-10-11 17:00:36 UTC
Red Hat Product Errata RHBA-2020:2580 0 None None None 2020-06-23 00:57:39 UTC

Description Dan Williams 2020-05-11 19:07:32 UTC
+++ This bug was initially created as a clone of Bug #1834473 +++

Multiple northds run for HA in active/passive mode where the active
northd holds a lock. If that northd loses connectivity to the database
or is killed without releasing the lock, ovsdb-server will clear
the lock after twice the inactivity probe. But if that probe is set
to 0 (disabled) that will never happen, and a new northd will never
grab the lock and continue reconciling NB->SB.

Set the DB inactivity probe to something greater than 0 to ensure
that a northd will always eventually become active. The value of 60 was
chosen as a reasonable middle-ground between the lock being cleared
and another northd grabbing it (~120s) and the possibility that a loaded
ovsdb-server (many ovn-controller clients) would take more than 30-40
seconds to send/reply to all inactivity probes from clients.

Comment 3 Ross Brattain 2020-06-17 05:56:48 UTC
Verified on 4.4.0-0.nightly-2020-06-15-203705

w9h72 has lock

northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:40Z|10355|jsonrpc|DBG|ssl: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:41Z|10372|jsonrpc|DBG|ssl: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:54Z|10429|jsonrpc|DBG|ssl: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"

w9h72: pkill -STOP ovn-northd

ccd5d takes lock ~120 seconds later

northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14663|jsonrpc|DBG|ssl: received notification, method="locked", params=["ovn_northd"]
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14664|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14674|jsonrpc|DBG|ssl: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:43:10Z|14739|jsonrpc|DBG|ssl: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
northd-ovnkube-master-ccd5d-northd:2020-06-17T05:43:11Z|14754|jsonrpc|DBG|ssl: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"

Comment 5 errata-xmlrpc 2020-06-23 00:57:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.