+++ This bug was initially created as a clone of Bug #1834473 +++ Multiple northds run for HA in active/passive mode where the active northd holds a lock. If that northd loses connectivity to the database or is killed without releasing the lock, ovsdb-server will clear the lock after twice the inactivity probe. But if that probe is set to 0 (disabled) that will never happen, and a new northd will never grab the lock and continue reconciling NB->SB. Set the DB inactivity probe to something greater than 0 to ensure that a northd will always eventually become active. The value of 60 was chosen as a reasonable middle-ground between the lock being cleared and another northd grabbing it (~120s) and the possibility that a loaded ovsdb-server (many ovn-controller clients) would take more than 30-40 seconds to send/reply to all inactivity probes from clients.
Verified on 4.4.0-0.nightly-2020-06-15-203705 w9h72 has lock northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:40Z|10355|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert" northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:41Z|10372|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert" northd-ovnkube-master-w9h72-northd:2020-06-17T05:40:54Z|10429|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert" w9h72: pkill -STOP ovn-northd ccd5d takes lock ~120 seconds later northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14663|jsonrpc|DBG|ssl:10.0.195.93:9642: received notification, method="locked", params=["ovn_northd"] northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14664|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active. northd-ovnkube-master-ccd5d-northd:2020-06-17T05:42:59Z|14674|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert" northd-ovnkube-master-ccd5d-northd:2020-06-17T05:43:10Z|14739|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert" northd-ovnkube-master-ccd5d-northd:2020-06-17T05:43:11Z|14754|jsonrpc|DBG|ssl:10.0.195.93:9642: send request, method="transact", params=["OVN_Southbound",{"lock":"ovn_northd","op":"assert"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2580