Bug 1831778

Summary:	Pass Raft election-timer value through ovn-ctl
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	Anil Vishnoi <avishnoi>
Component:	OVN	Assignee:	Ben Pickard <bpickard>
Status:	CLOSED ERRATA	QA Contact:	Ehsan Elahi <eelahi>
Severity:	high	Docs Contact:
Priority:	medium
Version:	RHEL 7.7	CC:	avishnoi, bpickard, ctrautma, dcbw, dceara, mmichels
Target Milestone:	---	Keywords:	FutureFeature
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:	perfscale-ovn
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-02-24 17:47:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1831755
Bug Blocks:

Description Anil Vishnoi 2020-05-05 15:38:08 UTC

Description of problem:
Once the following bug is resolved and closed, we need to update the ovn-ctl to pass the election-timer parameter through ovn-ctl for run_sb_ovsdb and run_nb_ovsdb. 

https://bugzilla.redhat.com/show_bug.cgi?id=1831755

This is just to track the required work to support this end to end for OVN/OVN-kubernetes.


Actual results:
Currently ovn-ctl does not support setting of election-timer value at the raft cluster (sb/nb) at the start-up. User need to explictly configure it on RAFT cluster leader using ovs-appctl.

Expected results:
User should be able to pass election-timer to ovn-ctl for run_sb_ovsdb and run_nb_ovsdb options to set the default election-timer.

Additional info:

Comment 2 Dan Williams 2021-04-14 01:52:15 UTC

Proposed upstream patch to ovsdb-tool create-cluster:

http://patchwork.ozlabs.org/project/openvswitch/patch/398fb37978f8eb1c208d7b9209d0f4c615efaf1c.camel@redhat.com/

doesn't help existing clusters, but they should persist their timer once successfully set.

Comment 3 Dan Williams 2021-04-14 01:54:04 UTC

Will need a follow-up for OVN's ovn-ctl to pass the value through on cluster create too.

Comment 4 Dan Williams 2021-05-25 17:45:02 UTC

v3 posted for ovsdb-tool: http://patchwork.ozlabs.org/project/openvswitch/list/?series=245675

Still needs OVN component for ovn-ctl once merged.

Comment 5 Dan Williams 2021-07-01 14:56:12 UTC

v4 merged upstream to OVS 2.16: http://patchwork.ozlabs.org/project/openvswitch/patch/796297662d70498c459ffc97894e497067aee1b8.camel@redhat.com/

Still needs OVN side for ovn-ctl.

Comment 12 Ehsan Elahi 2022-01-18 11:09:12 UTC

[root@bz_1831778 ~]# rpm -qa |grep -E 'ovn|openvswitch'
openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn-2021-central-21.09.1-23.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
ovn-2021-host-21.09.1-23.el8fdp.x86_64
ovn-2021-21.09.1-23.el8fdp.x86_64

[root@bz_1831778 ~]# ovsdb-tool --help | grep "\--election-timer"
[root@bz_1831778 ~]#

[root@1831778 ~]# /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=42.42.42.1 --db-nb-create-insecure-remote=yes --db-sb-addr=42.42.42.1 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=42.42.42.1 --db-sb-cluster-local-addr=42.42.42.1 --ovn-northd-nb-db=tcp:42.42.42.1:6641 --ovn-northd-sb-db=tcp:42.42.42.1:6642 --db-nb-election-timer=3000 --db-sb-election-timer=3000 start_northd
ovsdb-tool does not support --election-timer ... failed!
[root@1831778 ~]#
 
<======== election-timer arg not supported in ovsdb-tool

Verified on:
[root@bz_1831778 ~]# rpm -qa |grep -E 'ovn|openvswitch'
ovn-2021-host-21.12.0-11.el8fdp.x86_64
openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn-2021-21.12.0-11.el8fdp.x86_64
ovn-2021-central-21.12.0-11.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch

[root@bz_1831778 ~]# ovsdb-tool --help | grep "\--election-timer"
 [--election-timer=ms] create-cluster DB CONTENTS LOCAL
[root@bz_1831778 ~]#

<======== election-timer arg is supported in ovsdb-tool

## create a cluster and start ovn-northd
[root@bz_1831778 ~]# ps aux | grep ovsdb
root     1323901  0.0  0.0  12136  1112 pts/0    S+   05:43   0:00 grep --color=auto ovsdb
[root@bz_1831778 ~]#
[root@bz_1831778 ~]# /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=42.42.42.1 --db-nb-create-insecure-remote=yes --db-sb-addr=42.42.42.1 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=42.42.42.1 --db-sb-cluster-local-addr=42.42.42.1 --ovn-northd-nb-db=tcp:42.42.42.1:6641 --ovn-northd-sb-db=tcp:42.42.42.1:6642 --db-nb-election-timer=3000 --db-sb-election-timer=3000 start_northd
Creating cluster database /etc/ovn/ovnnb_db.db             [  OK  ]
Starting ovsdb-nb                                          [  OK  ]
Waiting for OVN_Northbound to come up 2022-01-18T10:54:54Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting...
2022-01-18T10:54:54Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connected
                                                           [  OK  ]
Creating cluster database /etc/ovn/ovnsb_db.db             [  OK  ]
Starting ovsdb-sb                                          [  OK  ]
Waiting for OVN_Southbound to come up 2022-01-18T10:54:54Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2022-01-18T10:54:54Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connected
                                                           [  OK  ]
Starting ovn-northd                                        [  OK  ]
[root@bz_1831778 ~]#
[root@bz_1831778 ~]# ps aux | grep ovsdb
root     1324121  0.0  0.0  47896  1572 ?        Ss   05:54   0:00 ovsdb-server: monitoring pid 1324122 (healthy)
root     1324122  0.0  0.0 195872  6788 ?        Sl   05:54   0:00 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/var/run/ovn/ovnnb_db.sock --pidfile=/var/run/ovn/ovnnb_db.pid --unixctl=/var/run/ovn/ovnnb_db.ctl --detach --monitor --remote=db:OVN_Northbound,NB_Global,connections --private-key=db:OVN_Northbound,SSL,private_key --certificate=db:OVN_Northbound,SSL,certificate --ca-cert=db:OVN_Northbound,SSL,ca_cert --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers --remote=ptcp:6641:42.42.42.1 /etc/ovn/ovnnb_db.db
root     1324144  0.0  0.0  47896  1568 ?        Ss   05:54   0:00 ovsdb-server: monitoring pid 1324145 (healthy)
root     1324145  0.0  0.0 195888  6376 ?        Sl   05:54   0:00 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-sb.log --remote=punix:/var/run/ovn/ovnsb_db.sock --pidfile=/var/run/ovn/ovnsb_db.pid --unixctl=/var/run/ovn/ovnsb_db.ctl --detach --monitor --remote=db:OVN_Southbound,SB_Global,connections --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate --ca-cert=db:OVN_Southbound,SSL,ca_cert --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers --remote=ptcp:6642:42.42.42.1 /etc/ovn/ovnsb_db.db
root     1324176  0.0  0.0  12136  1140 pts/0    S+   05:56   0:00 grep --color=auto ovsdb
[root@bz_1831778 ~]#
[root@bz_1831778 ~]# cat /var/log/ovn/ovsdb-server-nb.log
2022-01-18T10:42:56.323Z|00003|vlog(monitor)|INFO|opened log file /var/log/ovn/ovsdb-server-nb.log
2022-01-18T10:42:56.323Z|00004|daemon_unix(monitor)|INFO|pid 1323686 died, exit status 0, exiting
2022-01-18T10:54:54.267Z|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-nb.log
2022-01-18T10:54:54.281Z|00002|raft|INFO|term 2: 5440206232 ms timeout expired, starting election
2022-01-18T10:54:54.281Z|00003|raft|INFO|term 2: elected leader by 1+ of 1 servers
2022-01-18T10:54:54.281Z|00004|raft|INFO|local server ID is 9c3d
2022-01-18T10:54:54.284Z|00005|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.16.3
2022-01-18T10:54:54.284Z|00006|raft|INFO|Election timer changed from 1000 to 3000
2022-01-18T10:55:04.286Z|00007|memory|INFO|6788 kB peak resident set size after 10.0 seconds
2022-01-18T10:55:04.286Z|00008|memory|INFO|atoms:35 cells:34 monitors:2 raft-log:4 sessions:1

<=========== raft election-timer changed from default value 1000 to 3000. 

[root@bz_1831778 ~]# cat /var/log/ovn/ovsdb-server-sb.log
2022-01-18T10:42:56.433Z|00003|vlog(monitor)|INFO|opened log file /var/log/ovn/ovsdb-server-sb.log
2022-01-18T10:42:56.433Z|00004|daemon_unix(monitor)|INFO|pid 1323709 died, exit status 0, exiting
2022-01-18T10:54:54.324Z|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-sb.log
2022-01-18T10:54:54.325Z|00002|raft|INFO|term 2: 5440206276 ms timeout expired, starting election
2022-01-18T10:54:54.325Z|00003|raft|INFO|term 2: elected leader by 1+ of 1 servers
2022-01-18T10:54:54.325Z|00004|raft|INFO|local server ID is d626
2022-01-18T10:54:54.327Z|00005|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.16.3
2022-01-18T10:54:54.328Z|00006|raft|INFO|Election timer changed from 1000 to 3000
2022-01-18T10:55:04.330Z|00007|memory|INFO|6376 kB peak resident set size after 10.0 seconds
2022-01-18T10:55:04.330Z|00008|memory|INFO|atoms:337 cells:295 monitors:2 raft-log:5 sessions:1 txn-history:1 txn-history-atoms:16

<=========== raft election-timer changed from default value 1000 to 3000.

####### setting timer out of range (e.g. 50ms or 700000) ######

[root@bz_1831778 ~]# /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=42.42.42.1 --db-nb-create-insecure-remote=yes --db-sb-addr=42.42.42.1 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=42.42.42.1 --db-sb-cluster-local-addr=42.42.42.1 --ovn-northd-nb-db=tcp:42.42.42.1:6641 --ovn-northd-sb-db=tcp:42.42.42.1:6642 --db-nb-election-timer=50 --db-sb-election-timer=50 start_northd
Creating cluster database /etc/ovn/ovnnb_db.db ovsdb-tool: ovsdb error: election timer must be between 100 and 600000, in msec.
                                                           [FAILED]
Starting ovsdb-nb ovsdb-server: I/O error: /etc/ovn/ovnnb_db.db: open failed (No such file or directory)
                                                           [FAILED]
ovn-nbctl: unix:/var/run/ovn/ovnnb_db.sock: database connection failed (No such file or directory)
Waiting for OVN_Northbound to come up 2022-01-18T11:03:21Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting...
2022-01-18T11:03:21Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:22Z|00003|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting...
2022-01-18T11:03:22Z|00004|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:22Z|00005|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: waiting 2 seconds before reconnect
2022-01-18T11:03:24Z|00006|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting...
2022-01-18T11:03:24Z|00007|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:24Z|00008|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: waiting 4 seconds before reconnect
2022-01-18T11:03:28Z|00009|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting...
2022-01-18T11:03:28Z|00010|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:28Z|00011|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: continuing to reconnect in the background but suppressing further logging
2022-01-18T11:03:51Z|00012|fatal_signal|WARN|terminating with signal 14 (Alarm clock)
/usr/share/openvswitch/scripts/ovs-lib: line 602: 1324612 Alarm clock             "$@"
                                                           [FAILED]
Creating cluster database /etc/ovn/ovnsb_db.db ovsdb-tool: ovsdb error: election timer must be between 100 and 600000, in msec.
                                                           [FAILED]
Starting ovsdb-sb ovsdb-server: I/O error: /etc/ovn/ovnsb_db.db: open failed (No such file or directory)
                                                           [FAILED]
ovn-sbctl: unix:/var/run/ovn/ovnsb_db.sock: database connection failed (No such file or directory)
Waiting for OVN_Southbound to come up 2022-01-18T11:03:51Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2022-01-18T11:03:51Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:52Z|00003|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2022-01-18T11:03:52Z|00004|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:52Z|00005|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: waiting 2 seconds before reconnect
2022-01-18T11:03:54Z|00006|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2022-01-18T11:03:54Z|00007|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:54Z|00008|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: waiting 4 seconds before reconnect
2022-01-18T11:03:58Z|00009|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2022-01-18T11:03:58Z|00010|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory)
2022-01-18T11:03:58Z|00011|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: continuing to reconnect in the background but suppressing further logging
2022-01-18T11:04:21Z|00012|fatal_signal|WARN|terminating with signal 14 (Alarm clock)
/usr/share/openvswitch/scripts/ovs-lib: line 602: 1324636 Alarm clock             "$@"
                                                           [FAILED]
OVN Northbound DB is not running ... failed!

Comment 14 errata-xmlrpc 2022-02-24 17:47:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0674