Description of problem: Once the following bug is resolved and closed, we need to update the ovn-ctl to pass the election-timer parameter through ovn-ctl for run_sb_ovsdb and run_nb_ovsdb. https://bugzilla.redhat.com/show_bug.cgi?id=1831755 This is just to track the required work to support this end to end for OVN/OVN-kubernetes. Actual results: Currently ovn-ctl does not support setting of election-timer value at the raft cluster (sb/nb) at the start-up. User need to explictly configure it on RAFT cluster leader using ovs-appctl. Expected results: User should be able to pass election-timer to ovn-ctl for run_sb_ovsdb and run_nb_ovsdb options to set the default election-timer. Additional info:
Proposed upstream patch to ovsdb-tool create-cluster: http://patchwork.ozlabs.org/project/openvswitch/patch/398fb37978f8eb1c208d7b9209d0f4c615efaf1c.camel@redhat.com/ doesn't help existing clusters, but they should persist their timer once successfully set.
Will need a follow-up for OVN's ovn-ctl to pass the value through on cluster create too.
v3 posted for ovsdb-tool: http://patchwork.ozlabs.org/project/openvswitch/list/?series=245675 Still needs OVN component for ovn-ctl once merged.
v4 merged upstream to OVS 2.16: http://patchwork.ozlabs.org/project/openvswitch/patch/796297662d70498c459ffc97894e497067aee1b8.camel@redhat.com/ Still needs OVN side for ovn-ctl.
[root@bz_1831778 ~]# rpm -qa |grep -E 'ovn|openvswitch' openvswitch2.15-2.15.0-53.el8fdp.x86_64 ovn-2021-central-21.09.1-23.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch ovn-2021-host-21.09.1-23.el8fdp.x86_64 ovn-2021-21.09.1-23.el8fdp.x86_64 [root@bz_1831778 ~]# ovsdb-tool --help | grep "\--election-timer" [root@bz_1831778 ~]# [root@1831778 ~]# /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=42.42.42.1 --db-nb-create-insecure-remote=yes --db-sb-addr=42.42.42.1 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=42.42.42.1 --db-sb-cluster-local-addr=42.42.42.1 --ovn-northd-nb-db=tcp:42.42.42.1:6641 --ovn-northd-sb-db=tcp:42.42.42.1:6642 --db-nb-election-timer=3000 --db-sb-election-timer=3000 start_northd ovsdb-tool does not support --election-timer ... failed! [root@1831778 ~]# <======== election-timer arg not supported in ovsdb-tool Verified on: [root@bz_1831778 ~]# rpm -qa |grep -E 'ovn|openvswitch' ovn-2021-host-21.12.0-11.el8fdp.x86_64 openvswitch2.15-2.15.0-53.el8fdp.x86_64 ovn-2021-21.12.0-11.el8fdp.x86_64 ovn-2021-central-21.12.0-11.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch [root@bz_1831778 ~]# ovsdb-tool --help | grep "\--election-timer" [--election-timer=ms] create-cluster DB CONTENTS LOCAL [root@bz_1831778 ~]# <======== election-timer arg is supported in ovsdb-tool ## create a cluster and start ovn-northd [root@bz_1831778 ~]# ps aux | grep ovsdb root 1323901 0.0 0.0 12136 1112 pts/0 S+ 05:43 0:00 grep --color=auto ovsdb [root@bz_1831778 ~]# [root@bz_1831778 ~]# /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=42.42.42.1 --db-nb-create-insecure-remote=yes --db-sb-addr=42.42.42.1 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=42.42.42.1 --db-sb-cluster-local-addr=42.42.42.1 --ovn-northd-nb-db=tcp:42.42.42.1:6641 --ovn-northd-sb-db=tcp:42.42.42.1:6642 --db-nb-election-timer=3000 --db-sb-election-timer=3000 start_northd Creating cluster database /etc/ovn/ovnnb_db.db [ OK ] Starting ovsdb-nb [ OK ] Waiting for OVN_Northbound to come up 2022-01-18T10:54:54Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting... 2022-01-18T10:54:54Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connected [ OK ] Creating cluster database /etc/ovn/ovnsb_db.db [ OK ] Starting ovsdb-sb [ OK ] Waiting for OVN_Southbound to come up 2022-01-18T10:54:54Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting... 2022-01-18T10:54:54Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connected [ OK ] Starting ovn-northd [ OK ] [root@bz_1831778 ~]# [root@bz_1831778 ~]# ps aux | grep ovsdb root 1324121 0.0 0.0 47896 1572 ? Ss 05:54 0:00 ovsdb-server: monitoring pid 1324122 (healthy) root 1324122 0.0 0.0 195872 6788 ? Sl 05:54 0:00 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/var/run/ovn/ovnnb_db.sock --pidfile=/var/run/ovn/ovnnb_db.pid --unixctl=/var/run/ovn/ovnnb_db.ctl --detach --monitor --remote=db:OVN_Northbound,NB_Global,connections --private-key=db:OVN_Northbound,SSL,private_key --certificate=db:OVN_Northbound,SSL,certificate --ca-cert=db:OVN_Northbound,SSL,ca_cert --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers --remote=ptcp:6641:42.42.42.1 /etc/ovn/ovnnb_db.db root 1324144 0.0 0.0 47896 1568 ? Ss 05:54 0:00 ovsdb-server: monitoring pid 1324145 (healthy) root 1324145 0.0 0.0 195888 6376 ? Sl 05:54 0:00 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-sb.log --remote=punix:/var/run/ovn/ovnsb_db.sock --pidfile=/var/run/ovn/ovnsb_db.pid --unixctl=/var/run/ovn/ovnsb_db.ctl --detach --monitor --remote=db:OVN_Southbound,SB_Global,connections --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate --ca-cert=db:OVN_Southbound,SSL,ca_cert --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers --remote=ptcp:6642:42.42.42.1 /etc/ovn/ovnsb_db.db root 1324176 0.0 0.0 12136 1140 pts/0 S+ 05:56 0:00 grep --color=auto ovsdb [root@bz_1831778 ~]# [root@bz_1831778 ~]# cat /var/log/ovn/ovsdb-server-nb.log 2022-01-18T10:42:56.323Z|00003|vlog(monitor)|INFO|opened log file /var/log/ovn/ovsdb-server-nb.log 2022-01-18T10:42:56.323Z|00004|daemon_unix(monitor)|INFO|pid 1323686 died, exit status 0, exiting 2022-01-18T10:54:54.267Z|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-nb.log 2022-01-18T10:54:54.281Z|00002|raft|INFO|term 2: 5440206232 ms timeout expired, starting election 2022-01-18T10:54:54.281Z|00003|raft|INFO|term 2: elected leader by 1+ of 1 servers 2022-01-18T10:54:54.281Z|00004|raft|INFO|local server ID is 9c3d 2022-01-18T10:54:54.284Z|00005|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.16.3 2022-01-18T10:54:54.284Z|00006|raft|INFO|Election timer changed from 1000 to 3000 2022-01-18T10:55:04.286Z|00007|memory|INFO|6788 kB peak resident set size after 10.0 seconds 2022-01-18T10:55:04.286Z|00008|memory|INFO|atoms:35 cells:34 monitors:2 raft-log:4 sessions:1 <=========== raft election-timer changed from default value 1000 to 3000. [root@bz_1831778 ~]# cat /var/log/ovn/ovsdb-server-sb.log 2022-01-18T10:42:56.433Z|00003|vlog(monitor)|INFO|opened log file /var/log/ovn/ovsdb-server-sb.log 2022-01-18T10:42:56.433Z|00004|daemon_unix(monitor)|INFO|pid 1323709 died, exit status 0, exiting 2022-01-18T10:54:54.324Z|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-sb.log 2022-01-18T10:54:54.325Z|00002|raft|INFO|term 2: 5440206276 ms timeout expired, starting election 2022-01-18T10:54:54.325Z|00003|raft|INFO|term 2: elected leader by 1+ of 1 servers 2022-01-18T10:54:54.325Z|00004|raft|INFO|local server ID is d626 2022-01-18T10:54:54.327Z|00005|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.16.3 2022-01-18T10:54:54.328Z|00006|raft|INFO|Election timer changed from 1000 to 3000 2022-01-18T10:55:04.330Z|00007|memory|INFO|6376 kB peak resident set size after 10.0 seconds 2022-01-18T10:55:04.330Z|00008|memory|INFO|atoms:337 cells:295 monitors:2 raft-log:5 sessions:1 txn-history:1 txn-history-atoms:16 <=========== raft election-timer changed from default value 1000 to 3000. ####### setting timer out of range (e.g. 50ms or 700000) ###### [root@bz_1831778 ~]# /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=42.42.42.1 --db-nb-create-insecure-remote=yes --db-sb-addr=42.42.42.1 --db-sb-create-insecure-remote=yes --db-nb-cluster-local-addr=42.42.42.1 --db-sb-cluster-local-addr=42.42.42.1 --ovn-northd-nb-db=tcp:42.42.42.1:6641 --ovn-northd-sb-db=tcp:42.42.42.1:6642 --db-nb-election-timer=50 --db-sb-election-timer=50 start_northd Creating cluster database /etc/ovn/ovnnb_db.db ovsdb-tool: ovsdb error: election timer must be between 100 and 600000, in msec. [FAILED] Starting ovsdb-nb ovsdb-server: I/O error: /etc/ovn/ovnnb_db.db: open failed (No such file or directory) [FAILED] ovn-nbctl: unix:/var/run/ovn/ovnnb_db.sock: database connection failed (No such file or directory) Waiting for OVN_Northbound to come up 2022-01-18T11:03:21Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting... 2022-01-18T11:03:21Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:22Z|00003|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting... 2022-01-18T11:03:22Z|00004|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:22Z|00005|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: waiting 2 seconds before reconnect 2022-01-18T11:03:24Z|00006|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting... 2022-01-18T11:03:24Z|00007|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:24Z|00008|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: waiting 4 seconds before reconnect 2022-01-18T11:03:28Z|00009|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connecting... 2022-01-18T11:03:28Z|00010|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:28Z|00011|reconnect|INFO|unix:/var/run/ovn/ovnnb_db.sock: continuing to reconnect in the background but suppressing further logging 2022-01-18T11:03:51Z|00012|fatal_signal|WARN|terminating with signal 14 (Alarm clock) /usr/share/openvswitch/scripts/ovs-lib: line 602: 1324612 Alarm clock "$@" [FAILED] Creating cluster database /etc/ovn/ovnsb_db.db ovsdb-tool: ovsdb error: election timer must be between 100 and 600000, in msec. [FAILED] Starting ovsdb-sb ovsdb-server: I/O error: /etc/ovn/ovnsb_db.db: open failed (No such file or directory) [FAILED] ovn-sbctl: unix:/var/run/ovn/ovnsb_db.sock: database connection failed (No such file or directory) Waiting for OVN_Southbound to come up 2022-01-18T11:03:51Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting... 2022-01-18T11:03:51Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:52Z|00003|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting... 2022-01-18T11:03:52Z|00004|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:52Z|00005|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: waiting 2 seconds before reconnect 2022-01-18T11:03:54Z|00006|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting... 2022-01-18T11:03:54Z|00007|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:54Z|00008|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: waiting 4 seconds before reconnect 2022-01-18T11:03:58Z|00009|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting... 2022-01-18T11:03:58Z|00010|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (No such file or directory) 2022-01-18T11:03:58Z|00011|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: continuing to reconnect in the background but suppressing further logging 2022-01-18T11:04:21Z|00012|fatal_signal|WARN|terminating with signal 14 (Alarm clock) /usr/share/openvswitch/scripts/ovs-lib: line 602: 1324636 Alarm clock "$@" [FAILED] OVN Northbound DB is not running ... failed!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0674