Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2044614

Summary:	[OVN SCALE] ovsdb-server: Backport fixes for DB compaction timings
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	Ilya Maximets <i.maximets>
Component:	ovsdb2.16	Assignee:	Ilya Maximets <i.maximets>
Status:	CLOSED ERRATA	QA Contact:	Jianlin Shi <jishi>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	FDP 22.A	CC:	ctrautma, jhsiao, jishi, ralongi, tredaelli
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openvswitch2.16-2.16.0-43.el8fdp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-30 16:28:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ilya Maximets 2022-01-24 20:07:58 UTC

This BZ is to track backporting of 2 following ovsdb fixes to openvswitch2.16:

* 339f97044 2021-12-13 | ovsdb: storage: Randomize should_snapshot checks when the minimum time passed. [Ilya Maximets]
* bf07cc9cd 2021-12-13 | raft: Only allow followers to snapshot. [Dumitru Ceara]

These are not strictly bug fixes (so they wasn't backported in upstream), but
important behavior changes that helps to avoid long delays in control plane
operations in high-scale OVN deployments when ovsdb-server decides to create
a snapshot.

Comment 1 OvS team 2022-01-29 01:10:10 UTC

* Fri Jan 28 2022 Ilya Maximets <i.maximets> - 2.16.0-43
- ovsdb: storage: Randomize should_snapshot checks when the minimum time passed. [RH git: abe61535ca] (#2044614)
commit 339f97044e3c2312fbb65b932fa14a181acf40d5
Author: Ilya Maximets <i.maximets>
Date: Mon Dec 13 16:43:33 2021 +0100

ovsdb: storage: Randomize should_snapshot checks when the minimum time passed.

Snapshots are scheduled for every 10-20 minutes. It's a random value
in this interval for each server. Once the time is up, but the maximum
time (24 hours) not reached yet, ovsdb will start checking if the log
grew a lot on every iteration. Once the growth is detected, compaction
is triggered.

OTOH, it's very common for an OVSDB cluster to not have the log growing
very fast. If the log didn't grow 2x in 20 minutes, the randomness of
the initial scheduled time is gone and all the servers are checking if
they need to create snapshot on every iteration. And since all of them
are part of the same cluster, their logs are growing with the same
speed. Once the critical mass is reached, all the servers will start
creating snapshots at the same time. If the database is big enough,
that might leave the cluster unresponsive for an extended period of
time (e.g. 10-15 seconds for OVN_Southbound database in a larger scale
OVN deployment) until the compaction completed.

Fix that by re-scheduling a quick retry if the minimal time already
passed. Effectively, this will work as a randomized 1-2 min delay
between checks, so the servers will not synchronize.

Scheduling function updated to not change the upper limit on quick
reschedules to avoid delaying the snapshot creation indefinitely.
Currently quick re-schedules are only used for the error cases, and
there is always a 'slow' re-schedule after the successful compaction.
So, the change of a scheduling function doesn't change the current
behavior much.

Signed-off-by: Ilya Maximets <i.maximets>
Acked-by: Han Zhou <hzhou>
Acked-by: Dumitru Ceara <dceara>

Reported-at: https://bugzilla.redhat.com/2044614
Signed-off-by: Ilya Maximets <i.maximets>

* Fri Jan 28 2022 Ilya Maximets <i.maximets> - 2.16.0-42
- raft: Only allow followers to snapshot. [RH git: 915efc8c00] (#2044614)
commit bf07cc9cdb2f37fede8c0363937f1eb9f4cfd730
Author: Dumitru Ceara <dceara>
Date: Mon Dec 13 20:46:03 2021 +0100

raft: Only allow followers to snapshot.

Commit 3c2d6274bcee ("raft: Transfer leadership before creating
snapshots.") made it such that raft leaders transfer leadership before
snapshotting. However, there's still the case when the next leader to
be is in the process of snapshotting. To avoid delays in that case too,
we now explicitly allow snapshots only on followers. Cluster members
will have to wait until the current election is settled before
snapshotting.

Given the following logs taken from an OVN_Southbound 3-server cluster
during a scale test:

S2 (follower):
19:08:00.870Z|raft|INFO|server 8b8d is leader for term 43

We see that the leader to be (S3) receives the leadership transfer,
initiates the election and immediately after starts a snapshot that
takes ~9.5 seconds. During this time, S2 votes for S3 electing it
as cluster leader but S3 doesn't effectively become leader until it
finishes snapshotting, essentially keeping the cluster without a
leader for up to ~9.5 seconds.

With the current change, S3 will delay compaction and snapshotting until
the election is finished.

The only exception is the case of single-node clusters for which we
allow the node to snapshot regardless of role.

Acked-by: Han Zhou <hzhou>
Signed-off-by: Dumitru Ceara <dceara>
Signed-off-by: Ilya Maximets <i.maximets>

Reported-at: https://bugzilla.redhat.com/2044614
Signed-off-by: Ilya Maximets <i.maximets>

Comment 4 Jianlin Shi 2022-03-09 08:31:29 UTC

Hi Ilya,

Could you give some suggestions on how to test the patch? maybe use ovs only or ovn. thanks

Comment 5 Ilya Maximets 2022-03-10 12:22:44 UTC

(In reply to Jianlin Shi from comment #4)
> Hi Ilya,
> 
> Could you give some suggestions on how to test the patch? maybe use ovs only
> or ovn. thanks

I'm not sure what would be a not very time consuming way to test.  The check
I can think of is following:

1. Start a clustered database (e.g. with 3 ovsdb servers).  It's probably easier
   to just start OVN with clustered Sb and Nb databases.

2. Wait 21+ minutes to make sure that all servers passed the first scheduled
   compaction time.

3. Verify that servers didn't try to compact, e.g. by looking for following
   messages in logs:
    Transferring leadership to write a snapshot.
    OVN_Southbound: Database compaction took 12601ms
   None of them should be in logs.

4. Start to add resources to the database growing database files up to 10+MB.

5. Check that all servers compacted their database files by looking for log
   messages.

With fixes applied, compactions on different database servers should be
triggered in wider time intervals, i.e. ideally they should not overlap.
Since randomness is involved, overlaps could still happen, but should be much
less likely.

Comment 6 Ilya Maximets 2022-03-11 15:50:03 UTC

Correction to the reproducer:

With the described order of operations the first compaction will likely
not be logged, because it takes way less than 1 second.  In order to
workaround it we need to start the cluster with already big database
(50 - 100 MB).  Alternatively, we may change the steps a bit:

1. Start the cluster.
2. Start to add resources to the database growing database files.
3. Check that all servers compacted their database files by looking for log
   messages.
4. Wait for 21+ minutes.
5. Verify that there was no compactions in last 21 minutes.
6. Start adding resources again (+50% more) and wait for the next compaction
   on all servers.
7. Check the time difference between that last compaction triggered on
   different servers.

Comment 7 Jianlin Shi 2022-03-12 01:17:06 UTC

tested with following steps:

1. start cluster on 3 server:
 server1:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl" 
ip_s=1.1.178.16
ip_c1=1.1.178.17
ip_c2=1.1.178.18

$ctl_cmd --db-nb-addr=$ip_s --db-nb-create-insecure-remote=yes \                                      
                        --db-sb-addr=$ip_s --db-sb-create-insecure-remote=yes \                       
                        --db-nb-cluster-local-addr=$ip_s --db-sb-cluster-local-addr=$ip_s \
                        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \           
                        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 start_northd
server2:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl"                                                              
ip_s=1.1.178.16                                                                                       
ip_c1=1.1.178.17                                                                                      
ip_c2=1.1.178.18                                                                                      
                                                                                                      
                                                                                                      
$ctl_cmd --db-nb-addr=$ip_c1 --db-nb-create-insecure-remote=yes \                                     
                        --db-sb-addr=$ip_c1 --db-sb-create-insecure-remote=yes \                      
                        --db-nb-cluster-local-addr=$ip_c1 --db-sb-cluster-local-addr=$ip_c1 \         
                        --db-nb-cluster-remote-addr=$ip_s --db-sb-cluster-remote-addr=$ip_s \         
                        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \           
                        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 start_northd
server3:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl"                                                              
ip_s=1.1.178.16                                                                                       
ip_c1=1.1.178.17                                                                                      
ip_c2=1.1.178.18                                                                                      
                                                                                                      
                                                                                                      
$ctl_cmd --db-nb-addr=$ip_c2 --db-nb-create-insecure-remote=yes \                                     
                        --db-sb-addr=$ip_c2 --db-sb-create-insecure-remote=yes \                      
                        --db-nb-cluster-local-addr=$ip_c2 --db-sb-cluster-local-addr=$ip_c2 \         
                        --db-nb-cluster-remote-addr=$ip_s --db-sb-cluster-remote-addr=$ip_s \         
                        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \           
                        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 start_northd

2. add mass of ls and lr:
ovn-nbctl ls-add public
        for m in `seq 0 3`;do                                                                         
                for n in `seq 1 99`;do
                ovn-nbctl lr-add r${i}                                                                
                ovn-nbctl lrp-add r${i} r${i}_public 00:de:ad:ff:$m:$n 172.16.$m.$n/16                
                ovn-nbctl lrp-add r${i} r${i}_s${i} 00:de:ad:fe:$m:$n 173.$m.$n.1/24                  
                ovn-nbctl lr-nat-add r${i} dnat_and_snat 172.16.${m}.$((n+100)) 173.$m.$n.2           
                ovn-nbctl lrp-set-gateway-chassis r${i}_public hv1                                    
                
                # s1
                ovn-nbctl ls-add s${i}                                                                
                
                # s1 - r1
                ovn-nbctl lsp-add s${i} s${i}_r${i}                                                   
                ovn-nbctl lsp-set-type s${i}_r${i} router                                             
                ovn-nbctl lsp-set-addresses s${i}_r${i} "00:de:ad:fe:$m:$n 173.$m.$n.1"               
                ovn-nbctl lsp-set-options s${i}_r${i} router-port=r${i}_s${i}                         
                
                # s1 - vm1                                                                            
                ovn-nbctl lsp-add s$i vm$i                                                            
                ovn-nbctl lsp-set-addresses vm$i "00:de:ad:01:$m:$n 173.$m.$n.2"                      
                ovn-nbctl lrp-add r$i r${i}_public 40:44:00:00:$m:$n 172.16.$m.$n/16                  
                
                ovn-nbctl lsp-add public public_r${i}                                                 
                ovn-nbctl lsp-set-type public_r${i} router                                            
                ovn-nbctl lsp-set-addresses public_r${i} router                                       
                ovn-nbctl lsp-set-options public_r${i} router-port=r${i}_public nat-addresses=router  
                let i++
                if [ $i -gt 300 ];then
                        break;                                                                        
                fi
                done
                if [ $i -gt 300 ];then                                                                
                        break;
                fi
        done 

3. wait sb to compact: grep compact /var/log/ovn*
4. sleep 21m
5. add more ls and lr:
ovn-nbctl ls-add public
i=301
        for m in `seq 4 9`;do
                for n in `seq 1 99`;do
                ovn-nbctl lr-add r${i}
                ovn-nbctl lrp-add r${i} r${i}_public 00:de:ad:ff:$m:$n 172.16.$m.$n/16
                ovn-nbctl lrp-add r${i} r${i}_s${i} 00:de:ad:fe:$m:$n 173.$m.$n.1/24
                ovn-nbctl lr-nat-add r${i} dnat_and_snat 172.16.${m}.$((n+100)) 173.$m.$n.2
                ovn-nbctl lrp-set-gateway-chassis r${i}_public hv1
                
                # s1
                ovn-nbctl ls-add s${i}
                
                # s1 - r1
                ovn-nbctl lsp-add s${i} s${i}_r${i}
                ovn-nbctl lsp-set-type s${i}_r${i} router
                ovn-nbctl lsp-set-addresses s${i}_r${i} "00:de:ad:fe:$m:$n 173.$m.$n.1"
                ovn-nbctl lsp-set-options s${i}_r${i} router-port=r${i}_s${i}
                
                # s1 - vm1
                ovn-nbctl lsp-add s$i vm$i
                ovn-nbctl lsp-set-addresses vm$i "00:de:ad:01:$m:$n 173.$m.$n.2"
                ovn-nbctl lrp-add r$i r${i}_public 40:44:00:00:$m:$n 172.16.$m.$n/16
                
                ovn-nbctl lsp-add public public_r${i}
                ovn-nbctl lsp-set-type public_r${i} router
                ovn-nbctl lsp-set-addresses public_r${i} router
                ovn-nbctl lsp-set-options public_r${i} router-port=r${i}_public nat-addresses=router
                let i++
                if [ $i -gt 700 ];then
                        break;
                fi
                done
                if [ $i -gt 700 ];then
                        break;
                fi
        done
6. wait sb to compact


result on openvswitch2.16-27:

[root@wsfd-advnetlab16 ovn]# grep compact *
ovsdb-server-sb.log:2022-03-11T15:59:06.793Z|00060|ovsdb|INFO|OVN_Southbound: Database compaction took 3853ms
ovsdb-server-sb.log:2022-03-11T16:39:33.053Z|00821|ovsdb|INFO|OVN_Southbound: Database compaction took 18215ms
[root@wsfd-advnetlab17 bz2044621]# grep compaction /var/log/ovn/*
/var/log/ovn/ovsdb-server-sb.log:2022-03-11T15:56:39.992Z|00036|ovsdb|INFO|OVN_Southbound: Database compaction took 4523ms
/var/log/ovn/ovsdb-server-sb.log:2022-03-11T16:39:32.363Z|00611|ovsdb|INFO|OVN_Southbound: Database compaction took 17521ms
[root@wsfd-advnetlab18 bz2044621]# grep compact /var/log/ovn/* 
/var/log/ovn/ovsdb-server-sb.log:2022-03-11T15:58:03.710Z|00040|ovsdb|INFO|OVN_Southbound: Database compaction took 3499ms
/var/log/ovn/ovsdb-server-sb.log:2022-03-11T16:39:36.169Z|00596|ovsdb|INFO|OVN_Southbound: Database compaction took 12655ms

the compact interval between every server is less than 5s

result on openvswitch2.16-58:

[root@wsfd-advnetlab16 bz2044621]# grep compaction /var/log/ovn/* 
/var/log/ovn/ovsdb-server-sb.log:2022-03-12T00:17:47.726Z|00072|ovsdb|INFO|OVN_Southbound: Database compaction took 3724ms
/var/log/ovn/ovsdb-server-sb.log:2022-03-12T01:06:07.925Z|01170|ovsdb|INFO|OVN_Southbound: Database compaction took 15864ms
[root@wsfd-advnetlab17 bz2044621]# grep compaction /var/log/ovn/*
/var/log/ovn/ovsdb-server-sb.log:2022-03-12T00:15:46.407Z|00034|ovsdb|INFO|OVN_Southbound: Database compaction took 3477ms
/var/log/ovn/ovsdb-server-sb.log:2022-03-12T01:06:37.476Z|01149|ovsdb|INFO|OVN_Southbound: Database compaction took 18402ms
[root@wsfd-advnetlab18 bz2044621]# grep compact /var/log/ovn/*
/var/log/ovn/ovsdb-server-sb.log:2022-03-12T00:17:12.703Z|00042|ovsdb|INFO|OVN_Southbound: Database compaction took 3389ms
/var/log/ovn/ovsdb-server-sb.log:2022-03-12T01:06:41.769Z|01132|ovsdb|INFO|OVN_Southbound: Database compaction took 16428ms

<=== the interval is larger

Comment 9 errata-xmlrpc 2022-03-30 16:28:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openvswitch2.16 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1146