Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1780719

Summary: measure the time needed by ovn-controller to resync to a new SB db
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Timothy Redaelli <tredaelli>
Component: openvswitch2.12Assignee: lorenzo bianconi <lorenzo.bianconi>
Status: CLOSED ERRATA QA Contact: ying xu <yinxu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: RHEL 8.0CC: ctrautma, jhsiao, jishi, kfida, lorenzo.bianconi, qding, ralongi, tredaelli
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.12-2.12.0-10.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1720653 Environment:
Last Closed: 2020-01-22 04:03:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nb db file none

Description Timothy Redaelli 2019-12-06 17:39:15 UTC
+++ This bug was initially created as a clone of Bug #1720653 +++

Additional info:
We are interested in the time that it takes ovn-controller to resync to a
different southbound database with substantially the same content.

So for example after creating a lot of logical switches and switch
ports like in the scale tests, copy the southbound database file (or
create an entirely new one with different switches and ports) and start
a new ovsdb-server on a different port on the server.  The change ovn-
remote in the local vswitch and see how long it takes for ovn-
controller to update the flows in OVS based on the new database.

Girish said that the buffers that ovn-controller used to download the
new database were pretty small and when they bumped the size from IIRC
4k -> 64k the download itself went quite a bit faster.

But in the end, it is not clear what the delay from (a) change ovn-
remote to (b) OVS flows are synchronized, is and if we need to look
into optimizing that.

Comment 2 Jianlin Shi 2020-01-09 12:38:37 UTC
steps:

server:

systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642            
cp ~/ovnnb_db.txt /var/lib/ovn/ovnnb_db.db -f                                                         
systemctl restart ovn-northd                                                                          
                                                                                                      

systemctl start openvswitch                                    
ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.30.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.30.25
systemctl restart ovn-controller
                                                               
                                                                          
ovn-nbctl clear logical_switch_port 253a3b8b-76c6-4af0-ad12-d1513cf0c6a3 options
ovn-nbctl clear logical_switch_port 4cdaae8b-b472-4ae4-8f26-fd0e2a375bc1 options
ovn-nbctl clear logical_switch_port 9c3ce12f-0373-4004-9a12-7c3b34ce0ae4 options
ovn-nbctl clear logical_switch_port 063a7db6-2f19-45c2-a3b2-8414b4c40d38 options
ovn-nbctl clear logical_switch_port 173224b0-9e29-4c49-8bdf-85a0f4297bff options
ovn-nbctl clear logical_switch_port 2403f99f-9be3-427b-9937-71776bc44b5d options

wait a while for sb generating on server after setup

client:

systemctl start openvswitch
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.30.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.30.26
systemctl restart ovn-controller
sleep 10                                      
                                                      
ovs-vsctl add-port br-int vm0 -- set interface vm0 type=internal     
ovs-vsctl set Interface vm0 external_ids:iface-id=253a3b8b-76c6-4af0-ad12-d1513cf0c6a3
                
ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal
ovs-vsctl set Interface vm1 external_ids:iface-id=4cdaae8b-b472-4ae4-8f26-fd0e2a375bc1
               
ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal
ovs-vsctl set Interface vm2 external_ids:iface-id=9c3ce12f-0373-4004-9a12-7c3b34ce0ae4
ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal
ovs-vsctl set Interface vm3 external_ids:iface-id=063a7db6-2f19-45c2-a3b2-8414b4c40d38
                                                                
ovs-vsctl add-port br-int vm4 -- set interface vm4 type=internal
ovs-vsctl set Interface vm4 external_ids:iface-id=173224b0-9e29-4c49-8bdf-85a0f4297bff
ovs-vsctl add-port br-int vm5 -- set interface vm5 type=internal
ovs-vsctl set Interface vm5 external_ids:iface-id=2403f99f-9be3-427b-9937-71776bc44b5d

for i in {1..60}                                                                      
do                                                              
        ovs-ofctl dump-flows br-int | wc -l                                           
        sleep 1                 
done                                                
                     
ovs-ofctl dump-flows br-int | wc -l
num_flow=$(ovs-ofctl dump-flows br-int | wc -l)                   
ovs-vsctl set open . external_ids:ovn-remote=tcp:20.0.30.11:6642  
ovs-ofctl del-flows br-int                              
systemctl restart ovn-controller     
ovs-ofctl dump-flows br-int | wc -l          
ovs-vsctl set open . external_ids:ovn-remote=tcp:20.0.30.25:6642
systemctl restart ovn-controller                      
date > $(date +%H-%M).log  
                                
while :                                       
do                                                    
        ovs-ofctl dump-flows br-int | wc -l                          
        num2_flow=$(ovs-ofctl dump-flows br-int | wc -l)             
        date    
        [ $num_flow -le $num2_flow ] && break                  
        sleep 1                                                
done



on openvswitch2.12.0-8:

start time:
Thu Jan  9 07:19:34 EST 2020                                                                          
   
flow finish time:                                                                                                   
Thu Jan  9 07:20:45 EST 2020                                                                          
                                                                                                      
[root@ibm-x3650m4-01 bz1720653]# rpm -qa | grep -E "openvswitch|ovn"
kernel-kernel-networking-openvswitch-ovn-common-1.0-7.noarch
ovn2.12-2.12.0-19.el8fdp.x86_64                                                                       
ovn2.12-host-2.12.0-19.el8fdp.x86_64                                                                  
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
openvswitch2.12-2.12.0-8.el8fdp.x86_64                                                                
ovn2.12-central-2.12.0-19.el8fdp.x86_64

on openvswitch2.12.0-12:

start time:
Thu Jan  9 07:34:24 EST 2020

flow finish time:
Thu Jan  9 07:35:29 EST 2020

it seems that there no much improvement form 8 to 12.

Comment 3 Jianlin Shi 2020-01-09 12:39:26 UTC
Created attachment 1650959 [details]
nb db file

Comment 4 lorenzo bianconi 2020-01-10 09:57:28 UTC
hi Jianlin Shi,

50-100ms delta time is aligned to what is expected since most of the time will be spent parsing/creating the json file

Comment 5 Jianlin Shi 2020-01-10 09:59:51 UTC
set VERIFIED per comment 4

Comment 7 errata-xmlrpc 2020-01-22 04:03:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:0172